I've written a python program which read a stdout of another process by the pipe redirection.
However, the program sucks in this line:
print "[Input Thread] ", self.inputPipe.readline(ii)
The error is IOError: [Errno 0] Error
I found the explanation of windows errno 0. It makes confused because it defined as:
The operation completed successfully.
Why does an operation completed successfully lead to an error?
The name can trick you but ERROR_SUCCESS actually means there was no error.
From https://msdn.microsoft.com/en-us/library/windows/desktop/ms681382.aspx:
ERROR_SUCCESS
0 (0x0)
The operation completed successfully.
I know this is kind of old but having spent a fair amount of time trying to find a complete answer without success. So I figured I'd share what I've figured out.
The complete answer of how this happens, is when the pInvoke method you called "fails" but not because of an error.
huh you think
For example lets say you need to unhook a windows hook, but it gets called twice due to a bit of spaghetti or a paranoid level of defensive programming in your object architecture.
// hook assigned earlier
// now we call our clean up code
if (NativeMethods.UnhookWindowsHookEx(HookHandle) == 0)
{
// method succeeds normally so we do not get here
Log.ReportWin32Error("Error removing hook", Marshal.GetLastWin32Error());
}
// other code runs, but the hook is never reattached,
// due to paranoid defensive program you call your clean up code twice
if (NativeMethods.UnhookWindowsHookEx(HookHandle) == 0)
{
// pInvoke method failed (return zero) because there was no hook to remove
// however there was no error, the hook was already gone thus ERROR_SUCCESS (0)
// is our last error
Log.ReportWin32Error("Error removing hook", Marshal.GetLastWin32Error());
}
The windows API can be tricky. Most likely, the error number was not properly retrieved by the second program you mentioned. It was either overwritten or not pulled in at all; so it defaulted to 0.
You didn't say what the other program was; but for example, in .net, it is easy to omit the 'SetLastError' flag when declaring your external calls.
[DllImport('kernel32.dll', SetLastError = true)]
https://www.medo64.com/2013/03/error-the-operation-completed-successfully/
Related
I am trying to change the user of a print job in the queue, as I want to create it on a service account but send the job to another users follow-me printing queue. I'm using the win32 module in python. Here is an example of my code:
from win32 import win32print
JOB_INFO_LEVEL = 2
pclExample = open("sample.pcl")
printer_name = win32print.GetDefaultPrinter()
hPrinter = win32print.OpenPrinter(printer_name)
try:
jobID = win32print.StartDocPrinter(hPrinter, 1, ("PCL Data test", None, "RAW"))
# Here we try to change the user by extracting the job and then setting it again
jobInfoDict = win32print.GetJob(hPrinter, jobID , JOB_INFO_LEVEL )
jobInfoDict["pUserName"] = "exampleUser"
win32print.SetJob(hPrinter, jobID , JOB_INFO_LEVEL , jobInfoDict , win32print.JOB_CONTROL_RESUME )
try:
win32print.StartPagePrinter(hPrinter)
win32print.WritePrinter(hPrinter, pclExample)
win32print.EndPagePrinter(hPrinter)
finally:
win32print.EndDocPrinter(hPrinter)
finally:
win32print.ClosePrinter(hPrinter)
The problem is I get an error at the win32print.SetJob() line. If JOB_INFO_LEVEL is set to 1, then I get the following error:
(1804, 'SetJob', 'The specified datatype is invalid.')
This is a known bug to do with how the C++ works in the background (Issue here).
If JOB_INFO_LEVEL is set to 2, then I get the following error:
(1798, 'SetJob', 'The print processor is unknown.')
However, this is the processor that came from win32print.GetJob(). Without trying to change the user this prints fine, so I'm not sure what is wrong.
Any help would be hugely appreciated! :)
EDIT:
Using Python 3.8.5 and Pywin32 303
At the beginning I thought it was a misunderstanding (I was also a bit skeptical about the bug report), mainly because of the following paragraph (which apparently seems to be wrong) from [MS.Docs]: SetJob function (emphasis is mine):
The following members of a JOB_INFO_1, JOB_INFO_2, or JOB_INFO_4 structure are ignored on a call to SetJob: JobId, pPrinterName, pMachineName, pUserName, pDrivername, Size, Submitted, Time, and TotalPages.
But I did some tests and ran into the problem. The problem is as described in the bug: filling JOB_INFO_* string members (which are LPTSTRs) with char* data.
Submitted [GitHub]: mhammond/pywin32 - Fix: win32print.SetJob sending ANSI to UNICODE API (and none of the 2 errors pops up). It was merged to main on 220331.
When testing the fix, I was able to change various properties of an existing job, I was amazed that it didn't have to be valid data (like below), I'm a bit curious to see what would happen when the job would be executed (as now I don't have a connection to a printer):
Change pUserName to str(random.randint(0, 10000)) to make sure it changes on each script run (PrintScreens taken separately and assembled in Paint):
Ways to go further:
Wait for a new PyWin32 version (containing this fix) to be released. This is the recommended approach, but it will also take more time (and it's unclear when it will happen)
Get the sources, either:
from main
from b303 (last stable branch), and apply the (above) patch(1)
build the module (.pyd) and copy it in the PyWin32's site-packages directory on your Python installation(s). Faster, but it requires some deeper knowledge, and maintenance might become a nightmare
Footnotes
#1: Check [SO]: Run / Debug a Django application's UnitTests from the mouse right click context menu in PyCharm Community Edition? (#CristiFati's answer) (Patching UTRunner section) for how to apply patches (on Win).
I wrote my first snakemake rule that uses a python script for processing files:
rule sanitize_labels:
input:
"data/raw/labels/rois_essence_31_10_2019_final.shp",
"data/raw/labels/pts_carte_auto_final.shp"
output:
"data/interim/labels/rois_essence_31_10_2019_final.csv",
"data/interim/labels/pts_carte_auto_final.csv"
params:
crs = 32189,
log = True
script:
"../../scripts/data/sanitize_labels.py"
It runs successfully for the first file, than stops with this message:
Waiting at most 5 seconds for missing files.
MissingOutputException in line 9 of E:\code\projects\essences\workflow\rules\pre
processing.smk:
Missing files after 5 seconds:
data/interim/labels/pts_carte_auto_final.csv
This might be due to filesystem latency. If that is the case, consider to increa
se the wait time with --latency-wait.
Removing output files of failed job sanitize_labels since they might be corrupte
d:
data/interim/labels/rois_essence_31_10_2019_final.csv
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: E:\code\projects\essences\.snakemake\log\2020-02-10T025157.458955.
snakemake.log
I tried swapping file order both in input and output; always only the first file gets processed.
In my python script, I refer to input and output as snakemake.input[0] and snakemake.output[0]. If I understand correctly snakemake.input[0] is assigned the current input in each call of the script (no matter what's the number of inputs in the rule). Same goes for snakemake.output[0]. Is that correct?
Do you have other hints at what can cause this error?
I'm running snakemake version 5.10.0 (installed as snakemake-minimal from bioconda channel.).
Thanks a lot for any hint.
Adding to Maarten's answer, once you have specified the generic rule he provided, you then request the final outputs you want as the rule 'all' as your first rule:
rule all:
input: expand("data/interim/labels/{name}.csv", name=DATASETS)
If you place the input directive in your generic sanitize_labels rule, it is no longer generic. Snakemake expands the input you provide to create the same rule as in your question.
Go through the tutorial again if it's still not clear. While you may think and write your rules from start to finish, snakemake evaluates from finish to start. You request the final outputs in all (as inputs) and snakemake decides what needs to be run. It's confusing at first, but just remember to request your final output in all and keep your rules generic.
I think you need to take a look again at the "idea" behind snakemake. Probably what you need is something like this:
rule sanitize_labels:
input:
"data/raw/labels/{name}.shp"
output:
"data/interim/labels/{name}.csv"
params:
crs = 32189,
log = True
script:
"../../scripts/data/sanitize_labels.py"
Where you do not exactly specify the filename, but you tell snakemake how it can generate a certain output from a certain input. In this case, if your script needs both data/interim/labels/rois_essence_31_10_2019_final.csv and data/interim/labels/pts_carte_auto_final.csv, Snakemake "understands" how to make these files, and it knows which inputs it needs.
My Python script uses an ADODB.Recordset object. I use an ADODB.Command object with a collection of ADODB.Parameter objects to update a record in the set. After that, I check the state of the recordset, and it was 1, which is adStateOpen. But when I call MyRecordset.Close(), I get an exception complaining that the operation is invalid in the set's current state. What state could an open recordset be in that would make it invalid to close it, and what can I do to fix it?
Code is scattered between a couple of files. I'll work on getting an illustration together.
Yes, that was the problem. Once I change the value of one of a recordset's ADODB.Field objects, I have to either update the recordset using ADODB.Recordset.Update() or call CancelUpdate().
The reason I'm going through all this rigarmarole of the ADODB.Command object is that ADODB.Recordset.Update() fails at random (or so it seems to me) times, complaining that "query-based update failed because row to update cannot be found". I've never been able to predict when that will happen or find a reliable way to keep it from happening. My only choice when that happens is to replace the ADODB.Recordset.Update() call with the construction of a complete update query and executing it using an ADODB.Connection or ADODB.Command object.
I'm working on some machine learning code and today I've lost about 6 hours because simple typo.
It was this:
numpy.empty(100,100)
instead of
numpy.empty([100,100])
As I'm not really used to numpy, so I forgot the brackets. The code happily crunched the numbers and at the end, just before saving results to disk, it crashed on that line.
Just to put things in perspective I code on remote machine in shell, so IDE is not really an option. Also I doubt IDE would catch this.
Here's what I already tried:
running pylint - well pylint kinda works. After I've disabled everything apart of errors and warnings, it even seem to be usefull. But pylint have serious issue with imported modules. As seen on official bug tracker devs know about it, but cannot/won't do anything about it. There is suggested workaround, but ignoring whole module, would not help in my case.
running pychecker - if I create code snippet with the mistake I made, the pychecker reports error - same error as python interpreter. However if I run pychecker on the actual source file (~100 LOC) it reported other errors (unused vars, unused imports, etc.); but the faulty numpy line was skipped.
At last I have tried pyflakes but it does even less checking than pychecker/pylint combo.
So is there any reliable method which can check code in advance? Without actually running it.
A language with stronger type checking would have been able to save you from this particular error, but not from errors in general. There are plenty of ways to go wrong that pass static type checking. So if you have computations that takes a long time, it makes sense to adopt the following strategies:
Test the code from end to end on small examples (that run in a few seconds or minutes) before running it on big data that will consume hours.
Structure long-running computations so that intermediate results are saved to files on disk at appropriate points in the computation. This means that when something breaks, you can fix the problem and restart the computation from the last save point.
Run the code from the interactive interpreter, so that in the event of an exception you are returned to the interactive session, giving you a chance of being able to recover the data using a post-mortem debugging session. For example, suppose I have some long-running computation:
def work(A, C):
B = scipy.linalg.inv(A) # takes a long time when A is big
return B.dot(C)
I run this from the interactive interpreter and it raises an exception:
>>> D = work(A, C)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "q22080243.py", line 6, in work
return B.dot(C)
ValueError: matrices are not aligned
Oh no! I forgot to transpose C! Do I have to do the inversion of A again? Not if I call pdb.pm:
>>> import pdb
>>> pdb.pm()
> q22080243.py(6)work()
-> return B.dot(C)
(Pdb) B
array([[-0.01129249, 0.06886091, ..., 0.08530621, -0.03698717],
[ 0.02586344, -0.04872148, ..., -0.04853373, 0.01089163],
...,
[-0.11463087, 0.15048804, ..., 0.0722889 , -0.12388141],
[-0.00467437, -0.13650975, ..., -0.13894875, 0.02823997]])
Now, unlike in Lisp, I can't just set things right and continue the execution. But at least I can recover the intermediate results:
(Pdb) D = B.dot(C.T)
(Pdb) numpy.savetxt('result.txt', D)
Do you use unit tests? There is really no better way.
I am having an intermittent error causing my Python module to crash, and I'm assuming it's because of a memory error occurring by not getting the refcounts correct in the c code. I have a bit of code that gets a response at a random time from a remote location. Based on the data received, it needs to update a data variable which I should have access to in Python. What's the best way to accomplish this? The following code runs most of the time, and it works correctly when it does, but when it doesn't it crashes Python (bringing up the visual studio debug box). Thanks.
if (event == kResponseEvent) {
list = PyList_New(0);
for (i = 0; i < event->count; i++) {
PyList_Append(list, Py_BuildValue("{s:i, s:s}",
"id", event->id,
"name", event->name));
}
PyModule_AddObject(module, "names", list);
}
PyModule_AddObject() steals a reference. As such, you should not be decrefing list after.
PyList_New() can return NULL to indicate an error, which you aren't checking for. Py_BuildValue() can return NULL to indicate an error, which you aren't checking for. PyList_Append() can return -1 to indicate an error, which you're also not checking for. PyList_Append() doesn't steal the reference, so you're leaking the reference to the dict returned by Py_BuildValue(). The latter may be causing you to run out of memory, which can cause Py_BuildValue() or PyList_Append() to fail, and your failure to handle the error can cause a crash.
(Something else can also cause Py_BuildValue() or PyList_Append() to fail, but that's hard to guess at from just this snippet.)