Develop Reference

Decompress Python requests response with zlib

I'm trying to decompress the response from a web request using Python requests and zlib but I'm not able to decompress the response content properly. Here's my code:
import requests
import zlib
URL = "http://" #omitted real url
r = requests.get(URL)
print r.content
data = zlib.decompress(r.content, lib.MAX_WBITS)
print data
However, I keep getting various errors when changing the wbits parameter.
zlib.error: Error -3 while decompressing data: incorrect header check
zlib.error: Error -3 while decompressing data: invalid stored block lengths
I tried the wbits parameters for deflate, zlip and gzip as noted here zlib.error: Error -3 while decompressing: incorrect header check
But still can't get pass these errors. I'm trying to this in Python, I was given this piece of code that did it with Objective-C but I don't know Objective-C
#import "GTMNSData+zlib.h"
+ (NSData*) uncompress: (NSData*) data
{
Byte *bytes= (Byte*)[data bytes];
NSInteger length=[data length];
NSMutableData* retdata=[[NSMutableData alloc] initWithCapacity:length*3.5];
NSInteger bSize=0;
NSInteger offSet=0;
while (true) {
offSet+=bSize;
if (offSet>=length) {
break;
}
bSize=bytes[offSet];
bSize+=(bytes[offSet+1]<<8);
bSize+=(bytes[offSet+2]<<16);
bSize+=(bytes[offSet+3]<<24);
offSet+=4;
if ((bSize==0)||(bSize+offSet>length)) {
LogError(#"Invalid");
return data;
}
[retdata appendData:[NSData gtm_dataByInflatingBytes: bytes+offSet length:bSize]];
}
return retdata;
}

According to Python requests documentation at:
http://docs.python-requests.org/en/master/user/quickstart/#binary-response-content
it says:
You can also access the response body as bytes, for non-text requests:
>>> r.content
b'[{"repository":{"open_issues":0,"url":"https://github.com/...
The gzip and deflate transfer-encodings are automatically decoded for you.
If requests understand the encoding, it should therefore already be uncompressed.
Use r.raw if you need to get access to the original data to handle a different decompression mechanism.
http://docs.python-requests.org/en/master/user/quickstart/#raw-response-content

The following is an untested translation of the Objective-C code:
import zlib
import struct
def uncompress(data):
length = len(data)
ret = []
bSize = 0
offSet = 0
while True:
offSet += bSize
if offSet >= length:
break
bSize = struct.unpack("<i", data[offSet:offSet+4])
offSet += 4
if bSize == 0 or bSize + offSet > length:
print "Invalid"
return ''.join(ret)
ret.append(zlib.decompress(data[offSet:offSet+bSize]))
return ''.join(ret)

Windows pipes: Write from C - read in Python

I'd like to transmit a few bytes of data though a pipe to plot it from python.
I started with some snippets I found here but I cant get them working.
I've created the pipe like this:
int main(void){
HANDLE hPipe;
char buffer[24];
DWORD dwRead;
hPipe = CreateNamedPipe(TEXT("\\\\.\\pipe\\Pipe"),
PIPE_ACCESS_DUPLEX | PIPE_TYPE_BYTE | PIPE_READMODE_BYTE, // FILE_FLAG_FIRST_PIPE_INSTANCE is not needed but forces CreateNamedPipe(..) to fail if the pipe already exists...
PIPE_WAIT,
1,
24 * 16,
24 * 16,
NMPWAIT_USE_DEFAULT_WAIT,
NULL);
while (hPipe != INVALID_HANDLE_VALUE)
{
if (ConnectNamedPipe(hPipe, NULL) != FALSE) // wait for someone to connect to the pipe
{
while (ReadFile(hPipe, buffer, sizeof(buffer) - 1, &dwRead, NULL) != FALSE)
{
/* add terminating zero */
buffer[dwRead] = '\0';
/* do something with data in buffer */
printf("%s", buffer);
}
}
DisconnectNamedPipe(hPipe);
}
return 0;}
If I execute the following code it writes but the read part blocks:
import time
import struct
f = open(r'\\.\\pipe\\Pipe', 'r+b', 0)
i = 1
sss='ccccc'
while True:
s = sss.format(i)
i += 1
f.write(struct.pack('I', len(s)) + s) # Write str length and str
f.seek(0) # EDIT: This is also necessary
print 'Wrote:', s
n = struct.unpack('I', f.read(4))[0] # Read str length
s = f.read(n) # Read str
f.seek(0) # Important!!!
print 'Read:', s
time.sleep(2)
I tried commenting the ReadFile part in the C code but It did not work. Is there any other way to achieve this? I want to write from C and read from python. I tried writing into the pipe with CreateFile (from C) and it worked as expected. I only need the read part with python.

On most systems pipe is one-directional and you use two pipes to get two-directional (bidirectional) connection.
In your Python code you can open two connections
and then you don't need seek
import time
import struct
wf = open(r'Pipe', 'wb', 0)
rf = open(r'Pipe', 'rb', 0)
i = 0
template = 'Hello World {}'
while True:
i += 1
text = template.format(i)
# write text length and text
wf.write(struct.pack('I', len(text)))
wf.write(text)
print 'Wrote:', text
# read text length and text
n = struct.unpack('I', rf.read(4))[0]
read = rf.read(n)
print 'Read:', read
time.sleep(2)
EDIT: tested on Linux Mint 17, Python 3.4 & 2.7

I've solved it with PyWin32(http://sourceforge.net/projects/pywin32/files/) which seems to be the right tool for windows. I would rather use something more cross-plataform oriented but it has solved the problem.

Python Error: 'int' object has no attributes in dumps

I am getting the above error for the last two lines, and I am not sure what this error means. I am trying to get the "pretty" print in the output. Could someone please provide some insight about this error? What have I missed?
import json
import urllib
serviceurl = 'http://python-data.dr-chuck.net/comments_42.json'
while True:
url = serviceurl + urllib.urlencode({'sensor':'false', 'address': 'address'})
print "Retrieving", url
uh = urllib.urlopen(url)
data = uh.read()
print "Retrieved", len(data), "characters"
try: js = json.loads(str(data))
except: js = None
js = js["comments"][0]["count"]
print js.dumps(js, indent = 4)
print sum(js.get('count', 0) for js in js['comments'])
JSON data looks like this:
{
comments: [
{
name: "Matthias"
count: 97
},
{
name: "Geomer"
count: 97
}
...
]
}

You assigned an integer to the variable js on the line before:
js = js["comments"][0]["count"]
You can't then call js.dumps(), because int objects have no such method. Perhaps you wanted to call json.dumps() instead?
print json.dumps(js, indent = 4)
You'll have more problems however, as you replaced js with an int you also can't loop over it here:
print sum(js.get('count', 0) for js in js['comments'])
Perhaps you should use a different variable for the count; count would work, and comment in the loop:
count = js["comments"][0]["count"]
print json.dumps(js, indent = 4)
print sum(comment.get('count', 0) for comment in js['comments'])

Trouble with ReadProcessMemory in python to read 64bit process memory

I'm getting error code 998 (ERROR_NOACCESS) when using ReadProcessMemory to read the memory of a 64bit process (Minesweeper). I'm using python 3.5 64bit on windows 7 64bit.
The strange thing is, this error only happens with addresses that are higher up, like for example 0x 0000 0000 FF3E 0000. Lower addresses, like 0x 0000 0000 0012 AE40 don't throw an error and return correct Minesweeper data.
When I write the same program using nearly identical code in C#.NET and look at the same addresses, it works and I don't get an error!
I know the address I'm looking at is correct because I can see it with Cheat Engine and VMMap. I don't know if it's relevant, but the higher address I'm looking at is the base module address of the MineSweeper.exe module in Minesweeper.
Why is the python code not working?
Python code (throws error for higher addresses but works for lower):
import ctypes, struct
pid = 3484 # Minesweeper
processHandle = ctypes.windll.kernel32.OpenProcess(0x10, False, pid)
addr = 0x00000000FF3E0000 # Minesweeper.exe module base address
buffer = (ctypes.c_byte * 8)()
bytesRead = ctypes.c_ulonglong(0)
result = ctypes.windll.kernel32.ReadProcessMemory(processHandle, addr, buffer, len(buffer), ctypes.byref(bytesRead))
e = ctypes.windll.kernel32.GetLastError()
print('result: ' + str(result) + ', err code: ' + str(e))
print('data: ' + str(struct.unpack('Q', buffer)[0]))
ctypes.windll.kernel32.CloseHandle(processHandle)
# Output:
# result: 0, err code: 998
# data: 0
C#.NET code (64bit project, no errors):
[DllImport("kernel32.dll")]
static extern IntPtr OpenProcess(int dwDesiredAccess, bool bInheritHandle, int dwProcessId);
[DllImport("kernel32.dll")]
static extern Int32 ReadProcessMemory(IntPtr hProcess, IntPtr lpBaseAddress, [Out] byte[] buffer, UInt32 size, out IntPtr lpNumberOfBytesRead);
[DllImport("kernel32.dll")]
static extern bool CloseHandle(IntPtr hObject);
static void Main(string[] args)
{
var pid = 3484; // Minesweeper
var processHandle = OpenProcess(0x10, false, pid);
var addr = 0x00000000FF3E0000; // Minesweeper.exe module base address
var buffer = new byte[8];
IntPtr bytesRead;
var result = ReadProcessMemory(processHandle, new IntPtr(addr), buffer, (uint)buffer.Length, out bytesRead);
Console.WriteLine("result: " + result);
Console.WriteLine("data: " + BitConverter.ToInt64(buffer, 0).ToString());
CloseHandle(processHandle);
Console.ReadLine();
}
// Output:
// result: 1
// data: 12894362189

It's good to be explicit when using ctypes. Setting argtypes and restype appropriately will help check number and type of arguments, just like the DllImport statements in C#.
Here's a pedantic example:
import ctypes as c
from ctypes import wintypes as w
pid = 4568 # Minesweeper
k32 = c.WinDLL('kernel32', use_last_error=True)
OpenProcess = k32.OpenProcess
OpenProcess.argtypes = w.DWORD,w.BOOL,w.DWORD
OpenProcess.restype = w.HANDLE
ReadProcessMemory = k32.ReadProcessMemory
ReadProcessMemory.argtypes = w.HANDLE,w.LPCVOID,w.LPVOID,c.c_size_t,c.POINTER(c.c_size_t)
ReadProcessMemory.restype = w.BOOL
CloseHandle = k32.CloseHandle
CloseHandle.argtypes = [w.HANDLE]
CloseHandle.restype = w.BOOL
processHandle = OpenProcess(0x10, False, pid)
addr = 0x00000000FF900000 # Minesweeper.exe module base address
data = c.c_ulonglong()
bytesRead = c.c_ulonglong()
result = ReadProcessMemory(processHandle, addr, c.byref(data), c.sizeof(data), c.byref(bytesRead))
e = c.get_last_error()
print('result: {}, err code: {}, bytesRead: {}'.format(result,e,bytesRead.value))
print('data: {:016X}h'.format(data.value))
CloseHandle(processHandle)
Output:
result: 1, err code: 0, bytesRead: 8
data: 0000000300905A4Dh
Also note that you can pass the address of a data variable instead of creating a buffer and unpacking it with the struct module.

See eryksun's comment, it fixed my problem! Changed 'addr' to 'ctypes.c_void_p(addr)' in the ReadProcessMemory call.

How to get parameter values of a callback function?

I am trying to make call for a callback function in python.
I have a dll present at a path say 'dllpath'.
This dll have a callback function stated below:
Function prototype:
ULONG SetByteTotalsCallback(tFNByteTotals pCallback,BYTE interval);
Parameter discription:
tFNByteTotals pCallback: mode-IN, Callback function pointer
BYTE interval:mode-IN, Interval in seconds
Callback Prototype:
void ByteTotalsCallback(ULONGLONG txTotalBytes, ULONGLONG rxTotalBytes );
I want to call the function SetByteTotalsCallback and I want to print the values of txTotalBytes and rxTotalBytes.
I tried with following code:
from ctypes import *
filepath = r"<path to dll>"
gdll = WinDLL(filepath)
tx = c_longlong
rx = c_longlong
pCallback = CFUNCTYPE(tx, rx)
def ByteTotalsCallback(t, r):
try:
print 'Printing tx and rx: '
#print 'Transmitted bytes: ',t[0]
#print 'Received bytes: ',r[0]
#return 0
except:
print 'Error...'
byteTotal_func =pCallback(ByteTotalsCallback)
SetByteTotalsCallback = gdll.SetByteTotalsCallback
try:
print 'Return of SetByteTotals: ',SetByteTotalsCallback(pCallback(ByteTotalsCallback), c_byte(128))
except:
print 'Error found: '
After executing the above code, I observe that the function ByteTotalsCallback is not getting called but SetByteTotalsCallback(pCallback(ByteTotalsCallback), c_byte(128))
got called and returned back successfully.
Can someone help me?
Thanks in advance.
Regards,
Geet