Develop Reference

Execute bytes string in Python from an .exe file [duplicate]

How can I load an exe file—stored as a base64 encoded string—into memory and execute it without writing it to disk?
The point being, to put some kind of control/password/serial system in place and compile it with py2exe. Then I could execute that embedded file when ever I want in my code.

All of the mechanisms Python has for executing a child process require a filename.
And so does the underlying CreateProcess function in the Win32 API, so there's not even an easy way around it by dropping down to that level.
There is a way to do this by dropping down to ZwCreateProcess/NtCreateProcess. If you know how to use the low-level NT API, this post should be all you need to understand it. If you don't… it's way too much to explain in an SO answer.
Alternatively, of course, you can create or use a RAM drive, or even simulate a virtual filesystem, but that's getting a little silly as an attempt to avoid creating a file.
So, the right answer is to write the exe to a file, then execute it. For example, something like this:
fd, path = tempfile.mkstemp(suffix='.exe')
code = base64.b64decode(encoded_code)
os.write(fd, code)
os.fchmod(fd, 0o711)
os.close(fd)
try:
result = subprocess.call(path)
finally:
os.remove(path)
This should work on both Windows and *nix, but it's completely untested, and will probably have bugs on at least one platform.
Obviously, if you want to execute it multiple times, don't remove it until you're done with it. Or just use some appropriate persistent directory, and write it only if it's missing or out of date.

encode exe :
import base64
#encode exe file in base64 data
with open("Sample.exe", 'rb') as f:
read_exe_to_basae64 = base64.b64encode(f.read())
#encoded data will be like (really big text, don't worry) for e.g.:
b'TVqQAAMAAAAEAAAA//8AALgAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyAAAAA4fug4AtAnNIbgBTM0hVGhpcyBwcm9ncmFtIGNhbm5vdCBiZSBydW4gaW4gRE9TIG1vZGUuDQ0KJAAAAAAAAAA9AHveeWEVjXlhFY15YRWN+n0bjXhhFY0QfhyNfmEVjZB+GI14YRWNUmljaHlhFY0AAAAAAAAAAAAAAA'
#decode exe file:
with open("Sample2.exe", 'wb') as f:
f.write(base64.b64decode(read_exe_to_basae64))
exe file will be created in folder. If you don't want users to see it, just decode it in any random folder and delete it after use.

Python FTPlib is creating files with unix clrf instead of windows cldf

I'm on a Windows PC and I'm trying to download files from an FTP. The files download fine, but the only issue when I open them up in Notepad is that it's displayed with a Unix (LF). I've tried a couple of different fixes to be able to get it to be a Windows (CRLF), but nothing is working. The file is a UTF-16-LE encoded file.
Here are two sources I looked at two fix this, but nothing:
How to correctly download files using ftplib so line breaks are added for windows
https://effbot.org/librarybook/ftplib.htm
My code is currently as follows:
def downloadFiles(self, files, localFolder):
with FTP(host=self.host, user=self.username, passwd=self.password) as ftp:
ftp.cwd(self.root)
for file in files:
with open(os.path.join(localFolder, file.fileName), 'w', newline=None) as f:
ftp.retrlines(f'RETR {file.fileName}', lambda line, file=f: file.write(line+'\n'))
I've tried the line+'\r\n, but it just adds an extra line space instead.
Anyone have any ideas of how to fix this?

If anyone has the issue in the future with a utf-16 file you just need to set the ftp encoding to utf-16. I was looking for an encoding option at the file level, but apparently you need to set it at the connection level.
with FTP(host=self.host, user=self.username, passwd=self.password) as ftp:
ftp.encoding = 'utf-16'
ftp.cwd(self.root)
for file in files:
with open(os.path.join(localFolder, file.fileName), 'w', encoding='utf-16') as f:
ftp.retrlines(f'RETR {file.fileName}', lambda line: f.write(line + '\n'))

There is a simple command line utility unix2dos.
You can use use unix2dos utility on the files after ftp.
Also if you are text editing the file, use Notepad++ .
With Notepad++ you can manage the file's newline format and its encoding as well.

How can I store a .CHM Help File as text with Python?

I am using Pastebin to store the code of my python program to keep it updated on several computers. I am now trying to similarly maintain an updated help window. I saw that I could use .chm files to keep a full help dialog in a single file, but the files do not translate to text well.
I used a sample .chm file from Microsoft, I opened the file ("Viewhlp.chm") with notepad and copied the text to Pastebin, and then used the script below to attempt to recreate the .chm file. This does not work. It gives a "cannot open the file" message when opening directly and is simply ignored with PyWin32.
Is there another single file format for help dialogs that I can load with python?
import urllib2, sys
helpUrl = "http://pastebin.com/raw.php?i=a8rF2i8a"
originalPath = "Viewhlp.chm"
newPath = "NewHlp.chm"
try:
helpData = urllib2.urlopen(helpUrl)
except urllib2.URLError:
sys.exit()
currentHelp = helpData.read()
with open(newPath, mode="wb") as helpFile:
helpFile.write(currentHelp)
# briefly display using PyWin32 or just open the chm files directly
import win32help
win32help.HtmlHelp(0, None, win32help.HH_INITIALIZE, None)
link = win32help.HH_AKLINK()
link.indexOnFail = 1
link.url = ""
link.msgText = ""
link.msgTitle = ""
link.window = ""
win32help.HtmlHelp(0, originalPath, win32help.HH_KEYWORD_LOOKUP, link)
win32help.HtmlHelp(0, newPath, win32help.HH_KEYWORD_LOOKUP, link)

Notepad won't display the non-printing characters properly. Probably the easiest thing to do would be to base64 encode the .chm, then open the encoded version in notepad before you copy it to pastebin. Then unencode it when you read it:
currentHelp = base64.b64decode(helpData.read())

One way I convert things/documents like this is by installing a "Generic / Text Only" printer on my Windows system, and then selecting it and picking the "print to file" option in the printing dialog that appears when I try to print something from the associated application.
This results in a plain text file with what would have been printed in it. There's probably some way to automate it, although I've never tried.

opening and writing a large binary file python

I have a homebrew web based file system that allows users to download their files as zips; however, I found an issue while dev'ing on my local box not present on the production system.
In linux this is a non-issue (the local dev box is a windows system).
I have the following code
algo = CipherType('AES-256', 'CBC')
decrypt = DecryptCipher(algo, cur_share.key[:32], cur_share.key[-16:])
file = open(settings.STORAGE_ROOT + 'f_' + str(cur_file.id), 'rb')
temp_file = open(temp_file_path, 'wb+')
data = file.read(settings.READ_SIZE)
while data:
dec_data = decrypt.update(data)
temp_file.write(dec_data)
data = file.read(settings.READ_SIZE)
# Takes a dump right here!
# error in cipher operation (wrong final block length)
final_data = decrypt.finish()
temp_file.write(final_data)
file.close()
temp_file.close()
The above code opens a file, and (using the key for the current file share) decrypts the file and writes it to a temporary location (that will later be stuffed into a zip file).
My issue is on the file = open(settings.STORAGE_ROOT + 'f_' + str(cur_file.id), 'rb') line. Since windows cares a metric ton about binary files if I don't specify 'rb' the file will not read to end on the data read loop; however, for some reason since I am also writing to temp_file it never completely reads to the end of the file...UNLESS i add a + after the b 'rb+'.
if i change the code to file = open(settings.STORAGE_ROOT + 'f_' + str(cur_file.id), 'rb+') everything works as desired and the code successfully scrapes the entire binary file and decrypts it. If I do not add the plus it fails and cannot read the entire file...
Another section of the code (for downloading individual files) reads (and works flawlessly no matter the OS):
algo = CipherType('AES-256', 'CBC')
decrypt = DecryptCipher(algo, cur_share.key[:32], cur_share.key[-16:])
file = open(settings.STORAGE_ROOT + 'f_' + str(cur_file.id), 'rb')
filename = smart_str(cur_file.name, errors='replace')
response = HttpResponse(mimetype='application/octet-stream')
response['Content-Disposition'] = 'attachment; filename="' + filename + '"'
data = file.read(settings.READ_SIZE)
while data:
dec_data = decrypt.update(data)
response.write(dec_data)
data = file.read(settings.READ_SIZE)
# no dumps to be taken when finishing up the decrypt process...
final_data = decrypt.finish()
temp_file.write(final_data)
file.close()
temp_file.close()
Clarification
The cipher error is likely because the file was not read in its entirety. For example, I have a 500MB file I am reading in at 64*1024 bytes at a time. I read until I receive no more bytes, when I don't specify b in windows it cycles through the loop twice and returns some crappy data (because python thinks it is interacting with a string file not a binary file).
When I specify b it takes 10-15 seconds to completely read in the file, but it does it succesfully, and the code completes normally.
When I am concurrently writing to another file as i read in from the source file (as in the first example) if I do not specify rb+ it displays the same behavior as not even specifying b which is, that it only reads a couple segments from the file before closing the handle and moving on, i end up with an incomplete file and the decryption fails.

I'm going to take a guess here:
You have some other program that's continually replacing the files you're trying to read.
On linux, this other program works by atomically replacing the file (that is, writing to a temporary file, then moving the temporary file to the path). So, when you open a file, you get the version from 8 seconds ago. A few seconds later, someone comes along and unlinks it from the directory, but that doesn't affect your file handle in any way, so you can read the entire file at your leisure.
On Windows, there is no such thing as atomic replacement. There are a variety of ways to work around that problem, but what many people do is to just rewrite the file in-place. So, when you open a file, you get the version from 8 seconds ago, start reading it… and then suddenly someone else blanks the file to rewrite it. That does affect your file handle, because they've rewritten the same file. So you hit an EOF.
Opening the file in r+ mode doesn't do anything to solve the problem, but it adds a new problem that hides it: You're opening the file with sharing settings that prevent the other program from rewriting the file. So, now the other program is failing, meaning nobody is interfering with this one, meaning this one appears to work.
In fact, it could be even more subtle and annoying than this. Later versions of Windows try to be smart. If I try to open a file while someone else has it locked, instead of failing immediately, it may wait a short time and try again. The rules for exactly how this works depend on the sharing and access you need, and aren't really documented anywhere. And effectively, whenever it works the way you want, it means you're relying on a race condition. That's fine for interactive stuff like dragging a file from Explorer to Notepad (better to succeed 99% of the time instead of 10% of the time), but obviously not acceptable for code that's trying to work reliably (where succeeding 99% of the time just means the problem is harder to debug). So it could easily work differently between r and r+ modes for reasons you will never be able to completely figure out, and wouldn't want to rely on if you could…
Anyway, if any variation of this is your problem, you need to fix that other program, the one that rewrites the file, or possibly both programs in cooperation, to properly simulate atomic file replacement on Windows. There's nothing you can do from just this program to solve it.*
* Well, you could do things like optimistic check-read-check and start over whenever the modtime changes unexpectedly, or use the filesystem notification APIs, or… But it would be much more complicated than fixing it in the right place.

Downloading text files with Python and ftplib.FTP from z/os

I'm trying to automate downloading of some text files from a z/os PDS, using Python and ftplib.
Since the host files are EBCDIC, I can't simply use FTP.retrbinary().
FTP.retrlines(), when used with open(file,w).writelines as its callback, doesn't, of course, provide EOLs.
So, for starters, I've come up with this piece of code which "looks OK to me", but as I'm a relative Python noob, can anyone suggest a better approach? Obviously, to keep this question simple, this isn't the final, bells-and-whistles thing.
Many thanks.
#!python.exe
from ftplib import FTP
class xfile (file):
def writelineswitheol(self, sequence):
for s in sequence:
self.write(s+"\r\n")
sess = FTP("zos.server.to.be", "myid", "mypassword")
sess.sendcmd("site sbd=(IBM-1047,ISO8859-1)")
sess.cwd("'FOO.BAR.PDS'")
a = sess.nlst("RTB*")
for i in a:
sess.retrlines("RETR "+i, xfile(i, 'w').writelineswitheol)
sess.quit()
Update: Python 3.0, platform is MingW under Windows XP.
z/os PDSs have a fixed record structure, rather than relying on line endings as record separators. However, the z/os FTP server, when transmitting in text mode, provides the record endings, which retrlines() strips off.
Closing update:
Here's my revised solution, which will be the basis for ongoing development (removing built-in passwords, for example):
import ftplib
import os
from sys import exc_info
sess = ftplib.FTP("undisclosed.server.com", "userid", "password")
sess.sendcmd("site sbd=(IBM-1047,ISO8859-1)")
for dir in ["ASM", "ASML", "ASMM", "C", "CPP", "DLLA", "DLLC", "DLMC", "GEN", "HDR", "MAC"]:
sess.cwd("'ZLTALM.PREP.%s'" % dir)
try:
filelist = sess.nlst()
except ftplib.error_perm as x:
if (x.args[0][:3] != '550'):
raise
else:
try:
os.mkdir(dir)
except:
continue
for hostfile in filelist:
lines = []
sess.retrlines("RETR "+hostfile, lines.append)
pcfile = open("%s/%s"% (dir,hostfile), 'w')
for line in lines:
pcfile.write(line+"\n")
pcfile.close()
print ("Done: " + dir)
sess.quit()
My thanks to both John and Vinay

Just came across this question as I was trying to figure out how to recursively download datasets from z/OS. I've been using a simple python script for years now to download ebcdic files from the mainframe. It effectively just does this:
def writeline(line):
file.write(line + "\n")
file = open(filename, "w")
ftp.retrlines("retr " + filename, writeline)

You should be able to download the file as a binary (using retrbinary) and use the codecs module to convert from EBCDIC to whatever output encoding you want. You should know the specific EBCDIC code page being used on the z/OS system (e.g. cp500). If the files are small, you could even do something like (for a conversion to UTF-8):
file = open(ebcdic_filename, "rb")
data = file.read()
converted = data.decode("cp500").encode("utf8")
file = open(utf8_filename, "wb")
file.write(converted)
file.close()
Update: If you need to use retrlines to get the lines and your lines are coming back in the correct encoding, your approach will not work, because the callback is called once for each line. So in the callback, sequence will be the line, and your for loop will write individual characters in the line to the output, each on its own line. So you probably want to do self.write(sequence + "\r\n") rather than the for loop. It still doesn' feel especially right to subclass file just to add this utility method, though - it probably needs to be in a different class in your bells-and-whistles version.

Your writelineswitheol method appends '\r\n' instead of '\n' and then writes the result to a file opened in text mode. The effect, no matter what platform you are running on, will be an unwanted '\r'. Just append '\n' and you will get the appropriate line ending.
Proper error handling should not be relegated to a "bells and whistles" version. You should set up your callback so that your file open() is in a try/except and retains a reference to the output file handle, your write call is in a try/except, and you have a callback_obj.close() method which you use when retrlines() returns to explicitly file_handle.close() (in a try/except) -- that way you get explict error handling e.g. messages "can't (open|write to|close) file X because Y" AND you save having to think about when your files are going to be implicitly closed and whether you risk running out of file handles.
Python 3.x ftplib.FTP.retrlines() should give you str objects which are in effect Unicode strings, and you will need to encode them before you write them -- unless the default encoding is latin1 which would be rather unusual for a Windows box. You should have test files with (1) all possible 256 bytes (2) all bytes that are valid in the expected EBCDIC codepage.
[a few "sanitation" remarks]
You should consider upgrading your Python from 3.0 (a "proof of concept" release) to 3.1.
To facilitate better understanding of your code, use "i" as an identifier only as a sequence index and only if you irredeemably acquired the habit from FORTRAN 3 or more decades ago :-)
Two of the problems discovered so far (appending line terminator to each character, wrong line terminator) would have shown up the first time you tested it.

Use retrlines of ftplib to download file from z/os, each line has no '\n'.
It's different from windows ftp command 'get xxx'.
We can rewrite the function 'retrlines' to 'retrlines_zos' in ftplib.py.
Just copy the whole code of retrlines, and chane the 'callback' line to:
...
callback(line + "\n")
...
I tested and it worked.

you want a lambda function and a callback. Like so:
def writeLineCallback(line, file):
file.write(line + "\n")
ftpcommand = "RETR {}{}{}".format("'",zOsFile,"'")
filename = "newfilename"
with open( filename, 'w' ) as file :
callback_lambda = lambda x: writeLineCallback(x,file)
ftp.retrlines(ftpcommand, callback_lambda)
This will download file 'zOsFile' and write it to 'newfilename'