Develop Reference

How to make python script over windbg to make it's use easier?

import sys
import subprocess
command = 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64 -y ' + sys.argv[1] + ' -i ' + sys.argv[2] + ' -z ' + sys.argv[3] + ' -c "!analyze" '
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
I tried this code, I am trying to take input of crash dump name and exe location and then I have to display user understandable crash analysis ouput.How to do that using python scripting? Is it easier with cpp scripting?

take input of crash dump name and exe location and then I have to display user understandable crash analysis ouput.
It seems you want to parse the text output of the !analyze command. You can do that, but you should be aware that this command can have a lot of different output.
Let me assume you're analyzing a user mode crash dump. In such a case, I would first run a few simpler commands to check whether you got a legit dump. You may try the following commands:
|| to check the dump type (should be "user")
| to get the name of the executable (should match your application)
lmvm <app> to check the version number of your executable
If everything is fine, you can go on:
.exr -1: distinguish between a crash and a hang. A 80000003 breakpoint is more likely a hang or nothing at all.
This may help you decide if you should run !analyze or !analyze -hang.
How to do that using Python scripting?
[...] \Windows Kits\10\Debuggers\x64 -y ' + [...]
This path contains backslashes, so you want to escape them or use an r-string like r"C:\Program Files (x86)\Windows Kits\10\...".
You should probably start an executable here to make it work. cdb.exe is the command line version of WinDbg.
command.split()
This will not only split the arguments, but also the path to the exectuable. Thus subprocess.popen() will try to an application called C:\Program which does not exist.
This could fail even more often, depending on the arguments with spaces in sys.argv[].
I suggest that you pass the options as they are:
command = r'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe'
arguments = [command]
arguments.extend(['-y', sys.argv[1]]) # Symbol path
arguments.extend(['-i', sys.argv[2]]) # Image path
arguments.extend(['-z', sys.argv[3]]) # Dump file
arguments.extend(['-c', '!analyze']) # Command(s) for analysis
process = subprocess.Popen(arguments, stdout=subprocess.PIPE)
Note that there's no split() involved, which could split in wrong position.
Side note: -i may not work as expected. If you receive the crash dump from clients, they may have a different version than the one you have on disk. Set up a proper symbol server to mitigate this.
Is it easier with CPP scripting?
It will be different, not easier.
Working example
This is a Python code that considers the above. It's still a bit hacky because of the delays etc. but there's no real indicator other than time and output for deciding when a command finished. This succeeds with Python 3.8 on a crash dump of Windows Explorer.
import subprocess
import threading
import time
import re
class ReaderThread(threading.Thread):
def __init__(self, stream):
super().__init__()
self.buffer_lock = threading.Lock()
self.stream = stream # underlying stream for reading
self.output = "" # holds console output which can be retrieved by getoutput()
def run(self):
"""
Reads one from the stream line by lines and caches the result.
:return: when the underlying stream was closed.
"""
while True:
line = self.stream.readline() # readline() will block and wait for \r\n
if len(line) == 0: # this will only apply if the stream was closed. Otherwise there is always \r\n
break
with self.buffer_lock:
self.output += line
def getoutput(self, timeout=0.1):
"""
Get the console output that has been cached until now.
If there's still output incoming, it will continue waiting in 1/10 of a second until no new
output has been detected.
:return:
"""
temp = ""
while True:
time.sleep(timeout)
if self.output == temp:
break # no new output for 100 ms, assume it's complete
else:
temp = self.output
with self.buffer_lock:
temp = self.output
self.output = ""
return temp
command = r'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe'
arguments = [command]
arguments.extend(['-y', "srv*D:\debug\symbols*https://msdl.microsoft.com/download/symbols"]) # Symbol path, may use sys.argv[1]
# arguments.extend(['-i', sys.argv[2]]) # Image path
arguments.extend(['-z', sys.argv[3]]) # Dump file
arguments.extend(['-c', ".echo LOADING DONE"])
process = subprocess.Popen(arguments, stdout=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
reader = ReaderThread(process.stdout)
reader.start()
result = ""
while not re.search("LOADING DONE", result):
result = reader.getoutput() # ignore initial output
def dbg(command):
process.stdin.write(command+"\r\n")
process.stdin.flush()
return reader.getoutput()
result = dbg("||")
if "User mini" not in result:
raise Exception("Not a user mode dump")
else:
print("Yay, it's a user mode dump")
result = dbg("|")
if "explorer" not in result:
raise Exception("Not an explorer crash")
else:
print("Yay, it's an Explorer crash")
result = dbg("lm vm explorer")
if re.search(r"^\s*File version:\s*10\.0\..*$", result, re.M):
print("That's a recent version for which we should analyze crashes")
else:
raise Exception("That user should update to a newer version before we spend effort on old bugs")
dbg("q")

if you don't want to use windbg which is a gui
use cdb.exe it is console mode windbg it will output all the results to terminal
here is a demo
F:\>cdb -c "!analyze -v;qq" -z testdmp.dmp | grep -iE "bucket|owner"
DEFAULT_BUCKET_ID: BREAKPOINT
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
FOLLOWUP_NAME: MachineOwner
BUCKET_ID: BREAKPOINT_ntdll!LdrpDoDebuggerBreak+30
BUCKET_ID_IMAGE_STR: ntdll.dll
BUCKET_ID_MODULE_STR: ntdll
BUCKET_ID_FUNCTION_STR: LdrpDoDebuggerBreak
BUCKET_ID_OFFSET: 30
BUCKET_ID_MODTIMEDATESTAMP: c1bb301
BUCKET_ID_MODCHECKSUM: 1f647b
BUCKET_ID_MODVER_STR: 10.0.18362.778
BUCKET_ID_PREFIX_STR: BREAKPOINT_
FAILURE_BUCKET_ID: BREAKPOINT_80000003_ntdll.dll!LdrpDoDebuggerBreak
Followup: MachineOwner
grep is a general purpose string parser
it is built-in in Linux
it is available for windows in several places
if in 32 bit you can use it from gnuwin32 package / Cygwin
if in 64 bit you can find it in git
you can use the native findstr.exe also
:\>dir /b f:\git\usr\bin\gr*
grep.exe
groups.exe
or in msys / mingw / Cygwin / wsl / third party clones /
:\>dir /b /s *grep*.exe
F:\git\mingw64\bin\x86_64-w64-mingw32-agrep.exe
F:\git\mingw64\libexec\git-core\git-grep.exe
F:\git\usr\bin\grep.exe
F:\git\usr\bin\msggrep.exe
F:\msys64\mingw64\bin\msggrep.exe
F:\msys64\mingw64\bin\pcregrep.exe
F:\msys64\mingw64\bin\x86_64-w64-mingw32-agrep.exe
F:\msys64\usr\bin\grep.exe
F:\msys64\usr\bin\grepdiff.exe
F:\msys64\usr\bin\msggrep.exe
F:\msys64\usr\bin\pcregrep.exe
or you can write your own simple string parser in python / JavaScript / typescript / c / c++ / ruby / rust / whatever
here is a sample python word lookup and repeat script
import sys
for line in sys.stdin:
if "BUCKET" in line:
print(line)
lets check this out
:\>dir /b *.py
pyfi.py
:\>cat pyfi.py
import sys
for line in sys.stdin:
if "BUCKET" in line:
print(line)
:\>cdb -c "!analyze -v ;qq" -z f:\testdmp.dmp | python pyfi.py
DEFAULT_BUCKET_ID: BREAKPOINT
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
BUCKET_ID: BREAKPOINT_ntdll!LdrpDoDebuggerBreak+30
BUCKET_ID_IMAGE_STR: ntdll.dll
BUCKET_ID_MODULE_STR: ntdll
BUCKET_ID_FUNCTION_STR: LdrpDoDebuggerBreak
BUCKET_ID_OFFSET: 30
BUCKET_ID_MODTIMEDATESTAMP: c1bb301
BUCKET_ID_MODCHECKSUM: 1f647b
BUCKET_ID_MODVER_STR: 10.0.18362.778
BUCKET_ID_PREFIX_STR: BREAKPOINT_
FAILURE_BUCKET_ID: BREAKPOINT_80000003_ntdll.dll!LdrpDoDebuggerBreak

Running vulture from a python script

I'm trying to find a way to run vulture (which finds unused code in python projects) inside a python script.
vulture documentation can be found here:
https://pypi.org/project/vulture/
Does anyone know how to do it?
The only way I know to use vulture is by shell commands.
I tried to tun the shell commands from the script, using module subprocess, something like this:
process = subprocess.run(['vulture', '.'], check=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,universal_newlines=True)
which I though would have the same effect as running the shell command "vulture ."
but it doesn't work.
Can anyone help?
Thanks

Vulture dev here.
The Vulture package exposes an API, called scavenge - which it uses internally for running the analysis after parsing command line arguments (here in vulture.main).
It takes in a list of Python files/directories. For each directory, Vulture analyzes all contained *.py files.
To analyze the current directory:
import vulture
v = vulture.Vulture()
v.scavenge(['.'])
If you just want to print the results to stdout, you can call:
v.report()
However, it's also possible to perform custom analysis/filters over Vulture's results. The method vulture.get_unused_code returns a list of vulture.Item objects - which hold the name, type and location of unused code.
For the sake of this answer, I'm just gonna print the name of all unused objects:
for item in v.get_unused_code():
print(item.name)
For more info, see - https://github.com/jendrikseipp/vulture

I see you want to capture the output shown at console:
Below code might help:
import tempfile
import subprocess
def run_command(args):
with tempfile.TemporaryFile() as t:
try:
out = subprocess.check_output(args,shell=True, stderr=t)
t.seek(0)
console_output = '--- Provided Command: --- ' + '\n' + args + '\n' + t.read() + out + '\n'
return_code = 0
except subprocess.CalledProcessError as e:
t.seek(0)
console_output = '--- Provided Command: --- ' + '\n' + args + '\n' + t.read() + e.output + '\n'
return_code = e.returncode
return return_code, console_output
Your expected output will be displayed in console_output
Link:
https://docs.python.org/3/library/subprocess.html

dna assembly with velvet; it runs velveth but doesn't regocinse velvetg

I have the above code to run velvet. I can run velveth with no problems but it is not recognising the parameter for velvetg. I have checked the documentation, and I cannot see anything different to what I have. when the programme reaches velveteg, I get the following messege: [0.000000] unknown option: -ins_length 500.
import glob, sys, os, subprocess
def velvet_project():
print 'starting_velvet'
#'this is the directory where I copied the two test files. H*, I realised this is a subdirectory *.gastq.gz, to process all the files with that extention'
folders = glob.glob('/home/my_name/fastqs_test/H*')
#print folders
for folder in folders:
print folder
#looking for fastqs in each folder
fastqs=glob.glob(folder + '/*.fastq.gz')
#print fastqs
strain_id = os.path.basename(folder)
output= '/home/my_name/velvet_results/' + strain_id + '_velvet'
if os.path.exists(output):
print 'velevet folder already exist'
else:
os.makedirs(output)
#cmd is a command line within the programme#
cmd=['velveth', output, '59', '-fastq.gz','shortPaired',fastqs[0],fastqs[1]]
#print cmd
my_file=subprocess.Popen(cmd)#I got this from the documentation.
my_file.wait()
print 'velveth has finished'
cmd_2=['velvetg', output, '-ins_length 500', '-exp_cov auto', '-scaffoding no']
print cmd_2
my_file_2=subprocess.Popen(cmd_2)
my_file_2.wait()
print "velvet has finished :)"
print 'start'
velvet_project()

Redirecting Output From a Program to a File with Python: Specific Bug

I've been trying to run a Java program and capture it's STDOUT output to a file from the Python script. The idea is to run test files through my program and check if it matches the answers.
Per this and this SO questions, using subprocess.call is the way to go. In the code below, I am doing subprocess.call(command, stdout=f) where f is the file I opened.
The resulted file is empty and I can't quite understand why.
import glob
test_path = '/path/to/my/testfiles/'
class_path = '/path/to/classfiles/'
jar_path = '/path/to/external_jar/'
test_pattern = 'test_case*'
temp_file = 'res'
tests = glob.glob(test_path + test_pattern) # find all test files
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# cd into directory where the class files are and run the program
command = 'cd {p} ; java -cp {cp} package.MyProgram {tc_p}'
.format(p=class_path,
cp=jar_path,
tc_p=test_path + tc)
# execute the command and direct all STDOUT to file
subprocess.call(command.split(), stdout=f, stderr=subprocess.STDOUT)
# diff is just a lambda func that uses os.system('diff')
exec_code = diff(answers[i], test_path + temp_file)
if exec_code == BAD:
scream(':(')

I checked the docs for subprocess and they recommended using subprocess.run (added in Python 3.5). The run method returns the instance of CompletedProcess, which has a stdout field. I inspected it and the stdout was an empty string. This explained why the file f I tried to create was empty.
Even though the exit code was 0 (success) from the subprocess.call, it didn't mean that my Java program actually got executed. I ended up fixing this bug by breaking down command into two parts.
If you notice, I initially tried to cd into correct directory and then execute the Java file -- all in one command. I ended up removing cd from command and did the os.chdir(class_path) instead. The command now contained only the string to run the Java program. This did the trick.
So, the code looked like this:
good_code = 0
# Assume the same variables defined as in the original question
os.chdir(class_path) # get into the class files directory first
for i, tc in enumerate(tests):
with open(test_path+temp_file, 'w') as f:
# run the program
command = 'java -cp {cp} package.MyProgram {tc_p}'
.format(cp=jar_path,
tc_p=test_path + tc)
# runs the command and redirects it into the file f
# stores the instance of CompletedProcess
out = subprocess.run(command.split(), stdout=f)
# you can access useful info now
assert out.returncode == good_code

Python/Batch: Use Python to press any key to continue

I am using a python script to automate a process involving batch files. These are batch files that are used for other applications and I am not allowed to edit them.
At the end of the batch file, it prompts the following:
"Press any key to continue ..."
How do I use python to recognize when this prompt appears, and how do I respond to it? I want to be able to close the file so I can run the next batch file.
Currently I have found the following solution, but it's terrible and makes me feel dirty inside:
#Run the batch file with parameter DIABFile
subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile])
#Sit here like an idiot until I'm confident the batch file is finished
time.sleep(4)
#Press any key
virtual_keystrokes.press('enter')
Any ideas?
Attempt #1
p = subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile],
bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
while p.poll() is None:
line = p.stdout.readline()
print(line)
if line.startswith('Press any key to continue'):
p.communicate('\r\n')
Resulted in the following output and error:
b'\r\n'
Traceback (most recent call last):
File "C:\workspace\Perform_QAC_Check\Perform_QAC_Check.py", line 341, in <module>
main()
File "C:\workspace\Perform_QAC_Check\Perform_QAC_Check.py", line 321, in main
run_setup_builderenv(sandboxPath, DIABFile)
File "C:\workspace\Perform_QAC_Check\Perform_QAC_Check.py", line 126, in run_setup_builderenv
if line.startswith('Press any key to continue'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
The process tried to write to a nonexistent pipe.
The part that seemed weirdest to me was that the startswith first arg must be bytes or a tuple of bytes, not str. I looked up the documentation and it definitely should be a string? tutorial of startswith
So I looked online and found this little bit.
The error message seems to be a bug in Python, as it is exactly the other way around. But still, no problems here, add after line #75 in indian.py
try:
line = line.decode()
except AttributeError:
pass
And so I did.
Attempt #2
p = subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile],
bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
while p.poll() is None:
line = p.stdout.readline()
print(line)
try:
line = line.decode()
if line.startswith('Press any key to continue'):
p.communicate('\r\n')
except AttributeError:
pass
Resulted in the following output:
b'\r\n'
b'Build Environment is created.\r\n'
b'\r\n'
b'Please Refer to the directory: C:/directory\r\n'
b'\r\n'
And then it hangs there... That is the last output before the "Please press any key to continue" should show up, but it never does.
Notes
I have since taken the second script and asked it to find "Please Refer", which it does. Unfortunately, then the script hangs again at the line:
p.communicate('\r\n')
Ending the program, again, prints the error:
The process tried to write to a nonexistent pipe.
Which I believe is related to this bug.
I can't imagine what I'm trying to do is THAT out of the ordinary. Since this is seemingly a little more complicated than expected I would like to say I am using XP and Python version 3.3.

Something like the following should work:
p = subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile],
bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
while p.poll() is None:
line = p.stdout.readline()
if line.startswith('Press any key to continue'):
p.communicate('\r\n')

You could parse the output of the subprocess and match on the "Press any key to continue" phrase to continue on.
See this thread: read subprocess stdout line by line especially what he posted as Update2
It might look like this:
import subprocess
proc = subprocess.Popen([path + '\\' + batchFile, path + '\\' + DIABFile],stdout=subprocess.PIPE)
for line in iter(proc.stdout.readline,''):
if (line.rstrip() == "Press any key to..":
break;

As stated in the OP, none of the solutions were solving the problem. So at the end the solution from Bryce solved the problem for me:
subprocess.call([path + '\\' + batchFile, path + '\\' + DIABFile], stdin=subprocess.DEVNULL)

The solution from this post worked for me:
Try to execute cmd.exe /c YourCmdFile < nul
YourCmdFile - full path to your batch script