I have a Python program that generates over 300 files and uses bcp to move them to MSSQL. There is a high level of concurrency as about 21 files are being generated and bcp'd in at the same time. Here is the critical part of the program:
cmd = ['bcp', self.bcptbl, 'IN', outfile, '-f', 'bcpfmt.fmt', '-m1', '-U', uid, '-S', self.srv, '-P', pwd]
subprocess.check_output(cmd)
Three batch threads go at a time, 7 sub-threads each, so 21 concurrent processes. At a random file bcp fails with error:
[Microsoft][SQL Server Native Client 11.0]Unable to open BCP host data-file
The error might have something to do with the way I create file before BCP is invoked:
with open(outfile, 'a') as outf:
proc = Popen('ext_prog.exe', stdin=PIPE, stdout=outf, stderr=PIPE)
_, err = proc.communicate(input='\n'.join(patterns).encode('latin1'))
Something tells me that the file handle is not released by the external program, even though file open and close is seemingly handled by me.
This is not a typical error, as permissions, folders, paths, etc are all set up correctly, since it copies 80 ~ 150 files successfully before failing.
BCP call in the code above failed frequently until I inserted the following check before the bcp call:
#staticmethod
def wait_file_is_ready(outfile):
try:
with open(outfile, 'r'):
print("File {} is ready for reading".format(outfile))
except BaseException as e:
print("File {} is not ready: {}".format(outfile, e))
My reasoning is that Windows does not mark the file as closed in time so opening and closing it helps. This fixed 99% of errors but with the massive job I got today it came back to haunt me.
Things I tried to recover from error:
Adding a 1 hour sleep before re-running same bcp command - fails
Making a copy of the input file and re-running bcp command - fails
Running the BCP command manually from command line always works
More detailed code excerpt:
MAX_THREADS = 7
def start_batch(self):
ts = []
self.patternq = queue.Queue()
self.bcptbl = '"tempdb.dbo.outtbl_{}"'.format(randint(0,1E15))
for thread_no in range(MAX_THREADS):
tname = "thread_{:02}_of_{}".format(thread_no, MAX_THREADS)
t = Thread(name=tname, target=self.load, args=(thread_no,))
t.start()
ts.append(t)
for t in ts:
t.join()
def load(self, thread_no):
outfile = "d:\\tmp\\outfile_{}_{:02}.temp".format(
randint(0,1E15), thread_no)
try:
os.unlink(outfile)
except FileNotFoundError:
pass
while True:
try:
patterns = self.patternq.get_nowait()
except queue.Empty:
break
with open(outfile, 'a') as outf:
proc = Popen('ext_prog.exe', stdin=PIPE, stdout=outf, stderr=PIPE)
_, err = proc.communicate(input='\n'.join(patterns).encode('latin1'))
cmd = ['bcp', self.bcptbl, 'IN', outfile, '-f', 'bcpfmt.fmt', '-m1', '-U', uid, '-S', self.srv, '-P', pwd]
try:
subprocess.check_output(cmd)
except subprocess.CalledProcessError as e:
# OK, it failed because "Unable to open BCP host data-file"
# How can I recover from it?
raise
I went around the problem by using ODBC to insert the records, slowly and carefully. That worked 2 out of 3 times. Here is the error I got on third iteration:
os.unlink(outfile)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'd:\tmp\outfile_678410328373703.temp_03'
And a feasible explanation for the error:
Seems to be an issue long standing and a bug inherent with Windows 7
Found this in MS forums
Seems to be an issue long standing and a bug inherent with Windows 7.
There has been no "official" statement from MS acknowledging this bug hence a patch or fix release is unlikely. Some users in the thread above have offered "fixes" but they are way too time consuming and inefficient in regards to workflow productivity. This shouldn't be an issue one has to deal with when purchasing a new OS ...
Related
I have the following piece of code where the c++ executable (run.out) prints out a bunch of info in the runtime using std::cout. This code stores the outputs of run.out into the storage.txt.
storage = open("storage.txt", "w")
shell_cmd = "run.out"
proc = subprocess.Popen([shell_cmd], stdout=storage, stderr=storage)
Once the subprocess starts, I need to frequently check the contents of storage.txt and decide based on what has just been stored in there. How may I do that?
You could use subprocess.poll() which returns immediately and indicates if the subprocess is still running:
while proc.poll() is None:
time.sleep(0.25) # reads the content 4 times a seconds!
data = open("storage.txt").read()
if 'error' in data:
print("failed ...")
# somesomething ...
I'm nearly finished with a PyQt application for Windows, which I would like to have automatically update from a ZIP archive hosted on a remote server.
I have most of the update script complete, with it correctly downloading the new source and extracting it. My final step is to stop the software, replace the old source and restart the app.
My question is about which of the following is more appropriate:
to run the updater script through a system call to python and killing the software using python.
run the updater script using a system call to a batch file that kills the main software before overwriting it's source.
import the updater as a module and perform all of it's actions in the same process as the main software.
I can provide the scripts if necessary.
UPDATE:
So I've been exploring all the ways to do this including using multiprocessing (which produced a child process that was killed along with the parent) and subprocess.
The latter apparently can run the child process separately, which will allow me to shutdown the main app before extracting the new source. Here's what I have working:
#staticmethod
def install(folder):
# stop Pierre, unpack newest version, then restart Pierre.
try:
with open('pierre.pid', mode='r') as pid:
os.kill(int(pid.read()), signal.SIGINT)
with zipfile.ZipFile(file=folder) as zipped:
zipped.extractall(path='src')
try:
pierre = os.path.join(os.path.abspath(os.getcwd()), 'src/pierre.py')
exec(pierre)
except OSError as exc:
logging.error("Restarting Pierre failed. " + str(exc))
try:
os.remove('src.zip')
except OSError as exc:
logging.error("Deletion of zip failed. " + str(exc))
except zipfile.BadZipFile:
logging.error("Pierre update file is corrupt.")
except Exception as exc:
logging.error("Pierre update install failed. " + str(exc))
What's not working:
#staticmethod
def update_process():
# Begin the update process by spawning the updater script.
script = 'python ' + os.getcwd() + '\\updater.py'
subprocess.Popen([script])
The subprocess is producing a FileNotFoundError despite the path working when run manually in command prompt. (The second method is what starts the script, leading to the first method.)
I've got it. Here's what the process spawner looks like now:
#staticmethod
def update_process():
# Begin the update process by spawning the updater script.
script = os.path.join(os.getcwd() + '/updater.py')
script = script.replace('\\', '/')
subprocess.Popen([sys.executable, script], shell=True)
This launches the separate updater script.
I have the following code written in Python 2.7 on Windows. I want to check for updates for the current python script and update it, if there is an update, with a new version through ftp server preserving the filename and then executing the new python script after terminating the current through the os.kill with SIGNTERM.
I went with the exit function approach but I read that in Windows this only works with the atexit library and default python exit methods. So I used a combination of the atexit.register() and the signal handler.
***necessary libraries***
filematch = 'test.py'
version = '0.0'
checkdir = os.path.abspath(".")
dircontent = os.listdir(checkdir)
r = StringIO()
def exithandler():
try:
try:
if filematch in dircontent:
os.remove(checkdir + '\\' + filematch)
except Exception as e:
print e
ftp = FTP(ip address)
ftp.login(username, password)
ftp.cwd('/Test')
for filename in ftp.nlst(filematch):
fhandle = open(filename, 'wb')
ftp.retrbinary('RETR ' + filename, fhandle.write)
fhandle.close()
subprocess.Popen([sys.executable, "test.py"])
print 'Test file successfully updated.'
except Exception as e:
print e
ftp = FTP(ip address)
ftp.login(username, password)
ftp.cwd('/Test')
ftp.retrbinary('RETR version.txt', r.write)
if(r.getvalue() != version):
atexit.register(exithandler)
somepid = os.getpid()
signal.signal(SIGTERM, lambda signum, stack_frame: exit(1))
os.kill(somepid, signal.SIGTERM)
print 'Successfully replaced and started the file'
Using the:
signal.signal(SIGTERM, lambda signum, stack_frame: exit(1))
I get:
Traceback (most recent call last):
File "C:\Users\STiX\Desktop\Python Keylogger\test.py", line 50, in <module>
signal.signal(SIGTERM, lambda signum, stack_frame: exit(1))
NameError: name 'SIGTERM' is not defined
But I get the job done without a problem except if I use the current code in a more complex script where the script give me the same error but terminates right away for some reason.
On the other hand though, if I use it the correct way, signal.SIGTERM, the process goes straight to termination and the exit function never executed. Why is that?
How can I make this work on Windows and get the outcome that I described above successfully?
What you are trying to do seems a bit complicated (and dangerous from an infosec-perspective ;-). I would suggest to handle the reload-file-when-updated part of the functionality be adding a controller class that imports the python script you have now as a module and, starts it and the reloads it when it is updated (based on a function return or other technique) - look this way for inspiration - https://stackoverflow.com/a/1517072/1010991
Edit - what about exe?
Another hacky technique for manipulating the file of the currently running program would be the shell ping trick. It can be used from all programming languages. The trick is to send a shell command that is not executed before after the calling process has terminated. Use ping to cause the delay and chain the other commands with &. For your use case it could be something like this:
import subprocess
subprocess.Popen("ping -n 2 -w 2000 1.1.1.1 > Nul & del hack.py & rename hack_temp.py hack.py & hack.py ", shell=True)
Edit 2 - Alternative solution to original question
Since python does not block write access to the currently running script an alternative concept to solve the original question would be:
import subprocess
print "hello"
a = open(__file__,"r")
running_script_as_string = a.read()
b = open(__file__,"w")
b.write(running_script_as_string)
b.write("\nprint 'updated version of hack'")
b.close()
subprocess.Popen("python hack.py")
I'm working on a function that launches an external script using the subprocess module, the script runs sends/receives data files from a server, and does this for ~ 10,000 different files. The service occasionally gets hung up and can be continued if it gets relaunched. The external script saves the files (.xml) into a folder, so the function is complete when the number of input files equals the number of *.xml files in the output folder. This is what I have thus far, the second while loop seems to work - which is monitoring if there has been no files updated in the folder for 30 minutes, it should terminate the process. However, in the first while loop, after the process is terminated, it does not restart. Any help would be great!
from __future__ import division
import subprocess, datetime, time, os, glob
def runIPRscan(path, input, outputdir, email, num_complete):
num_files = len(glob.glob1(outputdir,"*.xml"))
while (num_files < num_complete):
#launch process
p = subprocess.Popen(['java', '-jar', path, '$#', '-i', input, '-m', email, '-o', outputdir], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
time.sleep(180) #give the script a few minutes to get running
while p.poll is None:
#wait 30s and check again
time.sleep(30)
num_files = len(glob.glob1(outputdir,"*.xml"))
if num_files == num_complete:
break
#monitor the output folder for recent changes in last 30 minutes
now = datetime.datetime.now()
ago = now - datetime.timedelta(minutes=30) #if nothing happens in 30 minutes, relaunch service
file_list = []
for path in glob.glob(outputdir + "/*.xml"):
(mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime) = os.stat(path)
if datetime.datetime.fromtimestamp(mtime) > ago:
file_list.append(path)
if not (file_list):
print("No activity in last 30 minutes, restarting service")
try:
p.terminate()
except OSError:
pass
time.sleep(10) #give it a few seconds to make sure process is closed before exiting
break
num_files = len(glob.glob1(outputdir,"*.xml"))
I finally figured it out. So it was actually a problem with the external script that was giving an error message to stderr but I couldn't see it because it was in the subprocess PIPE. So the above while loops are correct (I think), but I ended up replacing the external script with a python version and things seem to be working now. Thanks.
I'm trying to read command outputs from hcitools in Linux (it scans for bluetooth devices).
I just need to read the first line that it returns, as sometimes this tool has an error. The issue is that this tool continues to run in a infinite loop, which locks up the rest of my Python script. The script is run with sudo so that it has root privileges to use the hcitool command.
I have created a class to try to pipe the data in asynchronously:
class ASyncThread(threading.Thread): #pOpen read and readline are blocking. So we must use an async thread to read from hciTool
def __init__(self, command, parameters = []):
self.stdout = None
self.stderr = None
self.command = command
self.parameters = parameters
self.process = None
threading.Thread.__init__(self)
def run(self):
if len(self.command) >= 1:
self.process = subprocess.Popen([self.command] + self.parameters, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
self.stdout, self.stderr = self.process.communicate()
else:
print "[ASyncThread::run()] Error: Empty command given."
def terminate(self):
try:
self.process.terminate()
except Exception, ex:
print "[ASyncThread::terminate()] Error: ", ex
And I'm calling it with:
print "Checking HCI Tool Status..."
hciThread = ASyncThread("/usr/local/bin/hciconfig", ["lescan"])
hciThread.start()
time.sleep(1) #Give the program time to run.
hciThread.terminate() #If terminate is not placed here, it locks up my Python script when the thread is joined.
hciThread.join()
outputText = hciThread.stdout + " | " + hciThread.stderr
When this is run, the output is just " | ".
If I run this command:
sudo /usr/local/bin/hcitool lescan
It instantly starts working immediately:
slyke#ubuntu ~ $ sudo hcitool lescan
Set scan parameters failed: Input/output error
I've been working on this for a few hours now. I originally tried to do this with pOpen, but read() and readline() are both blocking. This is not normally a problem, except that there may not be an error, or any data produced by this command, so my Python script hangs. This is why I moved to threading, so it can wait for a second before stopping it, and continuing on.
It seems to me you cannot possibly join a thread, after you have just terminated it on the line above.
Your particular issue about doing an lescan is probably better solved with the solution from mikerr/btle-scan.py - https://gist.github.com/mikerr/372911c955e2a94b96089fbc300c2b5d