Develop Reference

How to determine error from earlier in DAG, in Airflow, for dependency/step with trigger rule "one_failed"

I currently have a DAG with two steps. The first step/dependency executes a function that pulls information in using the requests package and sends that info to a NodeJS server.
The second step/dependency is triggered with the rule "one_failed". However, I want to handle this situation differently depending on the type of error. There are two possible fail scenarios:
There could be an error when pulling in info using the requests package
There could be a timeout error when trying to connect to the NodeJS server
In the second step, that should only trigger upon a failure, I am wondering how I can go about determining what the error was from earlier in the workflow? Is it possible to pass the error info to the second step or to read the error info from within the second step using XCOMs? Any help would be greatly appreciated.

How to solve "Failed to set pipe buffer size"

I have a maybe somewhat naive question regarding pipe and pipe buffer...
I am not-advanced user of Ubuntu... And I am having a strange warning message in a python project at spyder...
/home/<myuser>/anaconda3/envs/spyder/lib/python3.9/site-packages/wurlitzer.py:209: RuntimeWarning: Failed to set pipe buffer size
warnings.warn("Failed to set pipe buffer size", RuntimeWarning)
I have not how to include a reproducible example here..., just the message... It is thrown in any code since started... but is just a warning and the codes appears to be running accordingly...
What that message means? And what should I do (or even what I have done wrong)?
I have not got reasonable result in a quick web search when I search the message associated with python or spider... I believe it can be related to Ubuntu because the questions like this one, this or this, so, not sure even if this is the correct place to make this question...
One of the packages I am using is Pool from multiprocessing (maybe it is related to multiprocessing)?...
I appreciate any help...

What does Disco's "Could not parse worker event:" error mean?

I'm trying to run a Disco job using map and reduce functions that are deserialized after being passed over a TCP socket using the marshal library. Specifically, I'm unpacking them with
code = marshal.loads(data_from_tcp)
func = types.FunctionType(code, globals(), "func")
I've already tested plain Disco jobs (with locally defined functions) on the same system, and they work fine. However, when I run a Disco job with the new functions, the jobs keep failing and I keep getting the error message localhost WARNING: [map:0] Could not parse worker event: invalid_length
I've searched the documentation, and there is no mention that I could find of a "worker event", or of an invalid_length. After doing a grep on the source code, I find a single instance of the phrase "Could not parse worker event:", specifically in the file master/src/disco_worker.erl. I'm not familiar with Erlang, and have no idea how this works.
What is causing this problem? Should I do something else to circumvent it?
EDIT: After more debugging, I've realized that this error is tied to my use of the string.split() method inside my test-case function. Whenever it is used (even on strings that are not part of the input), this error is raised. I've verified that the method does exist on the object, but calling it seems to cause problems. Any thoughts?
EDIT 2: In addition, any use of the re.split function achieves the same effect.
EDIT 3: It appears that calling any string function on the input string in the map function creates this same error.

In my case this warning occured always when I printed something to sys.stderr in map function (and the job failed in the end).
The documentation to worker protocol says: Workers should not write anything to stderr, except messages formatted as described below. stdout is also initially redirected to stderr.

Mysterious Apache Django WSGI Error

I'm running a django application on apache with mod_wsgi. The apache error log is clean except for one mysterious error.
Exception exceptions.TypeError: "'NoneType' object is not callable" in <bound method SharedSocket.__del__ of <httplib.SharedSocket instance at 0x82fb138c>> ignored
I understand that this means that some code is trying to call SharedSocket.del, but it is None. I have no idea what the reason for that is, or how to go about fixing it. Also, this error is marked as ignored. It also doesn't seem to be causing any actual problems other than filling the log file with error messages. Should I even bother trying to fix it?

It is likely that this is coming about because while handling an exception a further exception is occurring within the destructor of an object being destroyed and that the latter exception is unable to be raised because of the state of the pending one. Within Python C API details of such can be written direct to error log by PyErr_WriteUnraisable().
So, it isn't that the del method is None, but some variable it is trying to use from code executed within the del method is None. You would need to look at the code for SharedSocket.del to work out exactly what is going on.

Note: this is more of a pointer than an answer, but I couldn't get this to work in a comment.
I did some googling on the error message and there seems to be a group of related problems that crop up in Apache + mod_wsgi + MySQL environments. The culprit may be that you are running out of simultaneous connections to MySQL because of process pooling, with each process maintaining an open connection to the DB server. There are also indications that some non-thread-safe code may be used in a multi-thread environment. Good luck.

"Win32 exception occurred releasing IUnknown at..." error using Pylons and WMI

Im using Pylons in combination with WMI module to do some basic system monitoring of a couple of machines, for POSIX based systems everything is simple - for Windows - not so much.
Doing a request to the Pylons server to get current CPU, however it's not working well, or atleast with the WMI module. First i simply did (something) this:
c = wmi.WMI()
for cpu in c.Win32_Processor():
value = cpu.LoadPercentage
However, that gave me an error when accessing this module via Pylons (GET http://ip:port/cpu):
raise x_wmi_uninitialised_thread ("WMI returned a syntax error: you're probably running inside a thread without first calling pythoncom.CoInitialize[Ex]")
x_wmi_uninitialised_thread: <x_wmi: WMI returned a syntax error: you're probably running inside a thread without first calling pythoncom.CoInitialize[Ex] (no underlying exception)>
Looking at http://timgolden.me.uk/python/wmi/tutorial.html, i wrapped the code accordingly to the example under the topic "CoInitialize & CoUninitialize", which makes the code work, but it keeps throwing "Win32 exception occurred releasing IUnknown at..."
And then looking at http://mail.python.org/pipermail/python-win32/2007-August/006237.html and the follow up post, trying to follow that - however pythoncom._GetInterfaceCount() is always 20.
Im guessing this is someway related to Pylons spawning worker threads and crap like that, however im kinda lost here, advice would be nice.
Thanks in advance,
Anders
EDIT: If you are doing something similar, don't bother with the WMI module, simply use http://msdn.microsoft.com/en-us/library/aa394531%28VS.85%29.aspx , and you don't have to worry about threads crap like this.

Add "sys.coinit_flags = 0" after your "import sys" line and before the "import pythoncom" line. That worked for me, although I don't know why.

To me it sounds like Windows is not enjoying the way you are doing this kind of work on what are probably temporary worker threads (as you point out).
If this is the case, and you can't get things to work, one possible workaround would be to re-factor your application slightly so that there is a service thread running at all times which you can query for this information rather than setting everything up and asking for it on demand. It might not even need to be a thread, perhaps just a utility class instance which you get set up when the application starts, protected with a lock to prevent concurrent access.