begin TLDR;
I want to write a python3 script to scan through the memory of a running windows process and find strings.
end TLDR;
This is for a CTF binary. It's a typical Windows x86 PE file. The goal is simply to get a flag from the processes memory as it runs. This is easy with ProcessHacker you can search through the strings in the memory of the running application and find the flag with a regex. Now because I'm a masochistic geek I strive to script out solutions for CTFs (for everything really). Specifically I want to use python3, C# is also an option but would really like to keep all of the solution scripts in python.
Thought this would be a very simple task. You know... pip install some library written by someone that's already solved the problem and use it. Couldn't find anything that would let me do what I need for this task. Here are the libraries I tried out already.
ctypes - This was the first one I used, specifically ReadProcessMemory. Kept getting 299 errors which was because the buffer I was passing in was larger than that section of memory so I made a recursive function that would catch that exception, divide the buffer length by 2 until it got something THEN would read one byte at a time until it hit a 299 error. May have been on the right track there but I wasn't able to get the flag. I WAS able to find the flag only if I knew the exact address of the flag (which I'd get from process hacker). I may make a separate question on SO to address that, this one is really just me asking the community if something already exists before diving into this.
pymem - A nice wrapper for ctypes but had the same issues as above.
winappdbg - python2.x only. I don't want to use python 2.x.
haystack - Looks like this depends on winappdbg which depends on python 2.x.
angr - This is a possibility, Only scratched the surface with it so far. Looks complicated and it's on the to learn list but don't want to dive into something right now that's not going to solve the issue.
volatility - Looks like this is meant for working with full RAM dumps not for hooking into currently running processes and reading the memory.
My plan at the moment is to dive a bit more into angr to see if that will work, go back to pymem/ctypes and try more things. If all else fails ProcessHacker IS opensource. I'm not fluent in C so it'll take time to figure out how they're doing it. Really hoping there's some python3 library I'm missing or maybe I'm going about this the wrong way.
Ended up writing the script using the frida library. Also have to give soutz to rootbsd because his or her code in the fridump3 project helped greatly.
Related
I am having a Bus error in a Python script.
I could believe there is some issue with the memory there, but I don't know exactly what the source is.
I would expect it to be possible somehow to trace the line of Python code where this happens (even without a full stack).
It does say "core dumped", but no core is dumped, and I am not sure if Python core dumps can easily be used with gdb or the like to trace the line of code where the error happened.
What are my options? The error at the moment is cryptic in the sense that I don't know where the faulty memory access happens or why.
(I should mention that I did try to investigate for answers online before asking this, but didn't find anything useful. Just small pieces of "I am stuck" kind of things. I am guessing this error is very rare.)
EDIT:
$ python3 --version
Python 3.9.12
Yes, I am using C/C++ libraries, I believe, like numpy, copy (not sure if it is C/C++), torch.
I am not sure it is a good idea to post the code here, as it is quite a long .py file, and I am not sure exactly where the code gives that error, especially without a core dump actually being written to the disk.
I will mention that there are several parts where I am slightly concerned about unnecessary memory use that accumulates (this part is called a lot):
This loop:
for key in self.parameters.keys():
new = self.parameters[key]['x'].parameter
self.parser.parameters[key]['x'].parameter = new.detach().clone()
Before I buy my first setup. I'll launch my deep-learning-pipline on sth like vast.ai.
I never did it before, but How can I protect my script from being "stolen"?
This should be a serious launch and take around 7 days to finish training.
google colab doesn allow enough memory & ram for what i need ( need around 64GB ram)
is there a way to run a python script encrypted? (note: it makes use of libaries)
It is hard to run python encrypted. However, you could try to store the code into encrypted disk space.
There are some ways, from fully obfuscating your script to converting your script into equivalent cython or creating an executable out of it using the likes of Nuitka.
You may also implement some important logic in C/C++ (as extensions) and then call them in your script.
You may also set up a server where you feel is OK and send the bits that needs to be executed, basically create a distributed system.
As you can see there are many ways, and the deeper you go the more complex it gets.
Also you might want to have a look here as well.
How can I run c/c++ code within python in the form:
def run_c_code(code):
#Do something to run the code
code = """
Arbitrary code
"""
run_c_code(code)
It would be great if someone could provide an easy solution which does not involve installing packages. I know that C is not a scripting language but it would be great if it could do a 'mini'-compile that is able to run the code into the console. The code should run as it would compiled normally but this needs to be able to work on the fly as the rest of the code runs it and if possible, run as fast as normal and be able to create and edit variables so that python can use it. If necessary, the code can be pre-compiled into the code = """something""".
Sorry for all the requirements but if you can make the c code run in python then that would be great. Thanks in advance for all the answers..
As somebody else already pointed out, to run C/C++ code from "within" Python, you'd have to write said C/C++ code into an own file, compile it correctly, and then execute that program from your Python code.
You can't just type one command, compile it, and execute it. You always have to have the whole "framework" set up. You can't compile a program when you haven't yet written the } that ends the class/function/statement 20 lines later on. At this point you'd already have to write the whole C/C++ program for it to work. It's simply not meant to be interpreted on the run, line by line. You can do that with python, bash/dash/batch, and a few others. But C/C++ definitely isn't one of them.
With those come several issues. Firstly, the C/C++ part probably needs data from the Python part. I don't know of any way of doing it in RAM alone (maybe there is one, but I don't know), so the Python part would have to write it into a file, the C/C++ part would read and process it, then put the processed data into another file, and then the Python part would have to read that and continue.
Which brings another point up. Here we're already getting into multi-threading territory, because the moment you execute that C/C++ program you're dealing with a second thread. So, somehow, you'd have to coordinate those programs so that the Python part only continues once the C/C++ part is done. Shouldn't be a huge problem to get running, but it can be a nightmare to performance and RAM if done wrongly.
Without knowing to what extent you use that program, I also like to add that C/C++ isn't platform-independent like Python. You'll have to compile that program for every single different OS that you run it on. That may come with minor changes to the code and in general just a lot of work because you have to debug and test it for every single system.
To sum up, I think it may be better to find another solution. I don't know why you'd want to run this specific part in C/C++, but I'd recommend trying to get it done in one language. If there's absolutely no way you can get it done in Python (which I doubt, there's libraries for almost everything), you should get your Python to C/C++ instead.
If you want to run C/C++ code - you'll need either a C/C++ compiler, or a C/C++ interpreter.
The former is quite easy to arrange (though probably not suitable for an end user product) and you can just compile the code and run as required.
The latter requires that you attempt to process the code yourself and generate python code that you can then import. I'm not sure this one is worth the effort at all given that even websites that offer compilation tools wrap gcc/g++ rather than implement it in javascript.
I suspect that this is an XY problem; you may wish to take a couple of steps back and try to explain why you want to run c++ code from within a python script.
One of the most frustrating things about programming in Python thus far has been the lack of some kind of "pre-analysis". In Java, for example, a pre-analysis is performed before the actual compilation of a program, in which things like name usage is checked. In other words, if I have called a variable list_one in one area, and say I mispell it as list_on in another area, Java will say "Hey you cant do that, I dont know what list_on is."
Python does not seem to do this, and it is terribly frustrating! I have a program that takes about 15 minutes to run, and the last thing I was to see at 14.5 minutes into it is something like
NameError: name 'list_on' is not defined
Are their any tools available out there can can perform this kind of check before the interpreter actually runs the program? If not, what are some ways to work around this issue?
Have you considered checking your code with something like pyflakes or pylint?
UPDATE
I found a fantastic solution to this problem for those that happen to be emacs users. You can install PyFlakes-Flymake. This is a great tool! It will perform a static analysis of your code on the fly, and highlight trouble areas in red. I suggest using PIP instead of the suggested easy_install. Other than that, it is pretty simple to get it up and running. And well worth the effort!
I have a problem that I seriously spent months on now!
Essentially I am running code that requires to read from and save to HD5 files. I am using h5py for this.
It's very hard to debug because the problem (whatever it is) only occurs in like 5% of the cases (each run takes several hours) and when it gets there it crashes python completely so debugging with python itself is impossible. Using simple logs it's also impossible to pinpoint to the exact crashing situation - it appears to be very random, crashing at different points within the code, or with a lag.
I tried using OllyDbg to figure out whats happening and can safely conclude that it consistently crashes at the following location: http://i.imgur.com/c4X5W.png
It seems to be shortly after calling the python native PyObject_ClearWeakRefs, with an access violation error message. The weird thing is that the file is successfully written to. What would cause the access violation error? Or is that python internal (e.g. the stack?) and not file (i.e. my code) related?
Has anyone an idea whats happening here? If not, is there a smarter way of finding out what exactly is happening? maybe some hidden python logs or something I don't know about?
Thank you
PyObject_ClearWeakRefs is in the python interpreter itself. But if it only happens in a small number of runs, it could be hardware related. Things you could try:
Run your program on a different machine. if it doesn't crash there, it is probably a hardware issue.
Reinstall python, in case the installed version has somehow become corrupted.
Run a memory test program.
Thanks for all the answers. I ran two versions this time, one with a new python install and my same program, another one on my original computer/install, but replacing all HDF5 read/write procedures with numpy read/write procedures.
The program continued to crash on my second computer at odd times, but on my primary computer I had zero crashes with the changed code. I think it is thus safe to conclude that the problems were HDF5 or more specifically h5py related. It appears that more people encountered issues with h5py in that respect. Given that any error in my application translates to potentially large financial losses I decided to dump HDF5 completely in favor of other stable solutions.
Use a try catch statement. This can be put into the program in order to stop the program from crashing when erroneous data is entered