Develop Reference

How to deploy flow remotly on Prefect server?

I started working with the Prefect Orchestration tool.
My goal is to set up a server managing my automation on different other PCs and servers.
I do not fully understand the architecture of Prefect yet (with all these Agents etc.) but I managed to start a server on a remote Ubuntu environment.
To access the UI remotely I created a config.toml and added following lines:
[server]
endpoint = "<IPofserver>:4200/graphql"
[server.ui]
apollo_url = "http://<IPofserver>:4200/graphql"
[telemetry]
[server.telemetry]
enabled = false
The telemetry part is just to disable sending analysis data to Prefect.
Afterswards it was possible to accesss the UI from another PC and also to start an Agent on another PC with:
prefect agent local start --api "http://<IPofserver>:4200/graphql"
But how can I deploy flows now? A do not find an option to set their api like for the agent.
Even if I try to register a flow on the machine where the server itself is runnig I get following error message:
Traceback (most recent call last): File "", line 1, in
File
"/usr/local/lib/python3.10/dist-packages/prefect/core/flow.py", line
1726, in register
registered_flow = client.register( File "/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 831, in register
project = self.graphql(query_project).data.project # type: ignore File
"/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 443, in graphql
result = self.post( File "/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 398, in post
response = self._request( File "/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 633, in _request
response = self._send_request( File "/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 497, in _send_request
response = session.post( File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line
635, in post
return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py",
line 587, in request
resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line
695, in send
adapter = self.get_adapter(url=request.url) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line
792, in get_adapter
raise InvalidSchema(f"No connection adapters were found for {url!r}") requests.exceptions.InvalidSchema: No connection adapters
were found for ':4200/graphql'
Used Example Code:
import prefect
from prefect import task, Flow
#task
def say_hello():
logger = prefect.context.get("logger")
logger.info("Hello, Cloud!")
with Flow("hello-flow") as flow:
say_hello()
# Register the flow under the "tutorial" project
flow.register(project_name="Test")

If you are getting started with Prefect, I'd recommend using Prefect 2.0 - check this documentation page on getting started and this one about the underlying architecture.
If you still need help with Prefect Server and Prefect 1.0, check this extensive troubleshooting guide and if that doesn't help, send us a message on Slack, and we'll try to help you there.

Stackstorm 2.7 webhook "AccessRefused: 403" error

I have successfully created a webhook in stackstorm and it is visible in the webhook lists.
[centos#ip- ~]$ sudo st2 webhook list
+------------+------------------+-------------+
| url | type | description |
+------------+------------------+-------------+
| wfcreation | core.st2.webhook | |
+------------+------------------+-------------+
[centos#ip- ~]$
I triggered the webhook giving a payload and setting proper headers using stackstorm api key. The webhook gets triggered and returns with a status code of 200. But the underlying stackstorm workflow fails giving the below error.
{
"traceback": " File \"/opt/stackstorm/st2/lib/python2.7/site-packages/st2actions/container/base.py\", line 119, in _do_run
(status, result, context) = runner.run(action_params)
File \"/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py\", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File \"/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py\", line 206, in call
return attempt.get(self._wrap_exception)
File \"/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py\", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File \"/opt/stackstorm/st2/lib/python2.7/site-packages/retrying.py\", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File \"/opt/stackstorm/runners/mistral_v2/mistral_v2/mistral_v2.py\", line 247, in run
result = self.start_workflow(action_parameters=action_parameters)
File \"/opt/stackstorm/runners/mistral_v2/mistral_v2/mistral_v2.py\", line 284, in start_workflow
**options)
File \"/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/v2/executions.py\", line 65, in create
return self._create('/executions', data)
File \"/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/base.py\", line 100, in _create
self._raise_api_exception(resp)
File \"/opt/stackstorm/st2/lib/python2.7/site-packages/mistralclient/api/base.py\", line 160, in _raise_api_exception
error_message=error_data)
",
"error": "AccessRefused: 403"
}
The official stakstorm documentation does not have any reference of troubleshooting this error.
Any help would be greatly appreciated as I am blocked on this right now.

Finally I was able to figure out the problem was with the mistral-server service running on the stackstorm host.
The problem was that the mistral-server service was not able to connect to rabbitmq service due to a misconfiguration during stackstorm installation.But the misleading error message "AccessRefused: 403" did not pin point towards an issue with rabbitmq connection issue.
So here is what I found in the logs.
The error message in the mistral logs (/var/log/mistral/):
2018-06-25 15:30:19.309 10767 ERROR oslo_service.service AccessRefused: (0,0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
On digging the rabbitmq logs (/var/log/rabbitmq/):
=ERROR REPORT==== 25-Jun-2018::16:34:23 ===
closing AMQP connection <0.5118.0> (127.0.0.1:41248 -> 127.0.0.1:5672):
{handshake_error,starting,0,
{amqp_error,access_refused,
"AMQPLAIN login refused: user 'st2' - invalid credentials",
'connection.start_ok'}}
Clearly the st2 user credentials were wrongly configured during installation and that led to this whole issue.
Hope this helps somebody in future.

Another thing that could cause this is not having NGinx configured, and in that case the system wants you to use local ports. Here's an example that might help you: https://github.com/StackStorm/st2/tree/master/conf/st2.conf.sample and here's the nginx config.

dev-server HTTP Error 403: Forbidden

After updating from 1.7.5 (where everything worked fine) I'm getting a HTTP Error 403: Forbidden when trying to open any sites via localhost. Strange thing is I have pretty much the same setup at home as here at work and everything works there... Might be an issue with proxy server we're using at work, since that's the only difference I can think of? Here's the error log I'm getting, so if anyone knows what's going on please help (;
Traceback (most recent call last):
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 1302, in communicate
req.respond()
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 831, in respond
self.server.gateway(self).respond()
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 2115, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "U:\Dev\GAE\google\appengine\tools\devappserver2\wsgi_server.py", line 246, in __call__
return app(environ, start_response)
File "U:\Dev\GAE\google\appengine\tools\devappserver2\request_rewriter.py", line 311, in _rewriter_middleware
response_body = iter(application(environ, wrapped_start_response))
File "U:\Dev\GAE\google\appengine\tools\devappserver2\python\request_handler.py", line 89, in __call__
self._flush_logs(response.get('logs', []))
File "U:\Dev\GAE\google\appengine\tools\devappserver2\python\request_handler.py", line 220, in _flush_logs
apiproxy_stub_map.MakeSyncCall('logservice', 'Flush', request, response)
File "U:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 94, in MakeSyncCall
return stubmap.MakeSyncCall(service, call, request, response)
File "U:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 320, in MakeSyncCall
rpc.CheckSuccess()
File "U:\Dev\GAE\google\appengine\api\apiproxy_rpc.py", line 156, in _WaitImpl
self.request, self.response)
File "U:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 200, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "U:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 226, in _MakeRealSyncCall
encoded_response = self._server.Send(self._path, encoded_request)
File "U:\Dev\GAE\google\appengine\tools\appengine_rpc.py", line 393, in Send
f = self.opener.open(req)
File "U:\Dev\Python\lib\urllib2.py", line 410, in open
response = meth(req, response)
File "U:\Dev\Python\lib\urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "U:\Dev\Python\lib\urllib2.py", line 448, in error
return self._call_chain(*args)
File "U:\Dev\Python\lib\urllib2.py", line 382, in _call_chain
result = func(*args)
File "U:\Dev\Python\lib\urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
INFO 2013-04-19 12:28:52,576 server.py:561] default: "GET / HTTP/1.1" 500 -
INFO 2013-04-19 12:28:52,619 server.py:561] default: "GET /favicon.ico HTTP/1.1" 304 -
Also, the launcher throws an error when closing:
Traceback (most recent call last):
File "launcher\mainframe.pyc", line 327, in OnStop
File "launcher\taskcontroller.pyc", line 167, in Stop
File "launcher\dev_appserver_task_thread.pyc", line 82, in stop
File "launcher\taskthread.pyc", line 107, in stop
File "launcher\platform.pyc", line 397, in KillProcess
pywintypes.error: (5, 'TerminateProcess', 'Access is denied.')

I had this very same issue with my MacOSX when using a proxy server using Google App Engine Launcher 1.8.6. Apparently there's an issue with "proxy_bypass" on "urllib2.py".
There are two possible solutions:
Downgrade to 1.7.5, but, who wants to downgrade?
Edit "[GAE Instalattion path]/google/appengine/tools/appengine_rpc.py" and look for the line that says
opener.add_handler(fancy_urllib.FancyProxyHandler())
In my computer it was line 578, and then put a hash (#) at the beginning of the line, like this:
`#opener.add_handler(fancy_urllib.FancyProxyHandler())`
Save the file, stop and then restart your application. Now dev_appserver.py shouldn't try to use any proxy server at all.
If your application uses any external resources like a SOAP Webservice or something like that and you can't reach the server without the proxy server, then you'll have to downgrade. Please keep in mind that external javascript files (like facebook SDK or similar) are loaded from your browser, not from your application.
Since I'm not using any external REST or SOAP services it worked for me!
Hopefully it will work for you as well.

Try either:
-Accessing it through a different proxy. I.E a . proxy within a proxy
-Accessing it through your local IP i.e 192.168.1.1

I faced the same issue with version 1.9.5. Seems that the API proxy is sending some RPCs to the proxy server, which are then being rejected with HTTP 403 (since proxy servers are generally configured to reject connection attempts to arbitrary ports). In my case I was using the urlfetch module in my app to access external web pages, so disabling the proxy server was not a choice for me.
This is how I worked around the issue some time back (most probably it was based on comments found under this issue, but I cannot remember the exact sources).
NOTE:
For this approach to work, you'll have to know the hostname/IP address and default port of your proxy server, and change them appropriately in the code if you happen to connect to a different proxy server.
When you are not behind the proxy server, you will have to revert the applied changes in order to return to a working state (if you want internet access inside your app).
Here it goes:
Disable proxy settings for the Python (Google App Engine Launcher) environment in some way. (In my case it was easy since I was launching the dev_appserver.py from a Terminal shell (on Linux), and the unset http_proxy and unset https_proxy commands did the trick.)
Edit {App Engine SDK root}/google/appengine/api/urlfetch_stub.py. Find the code block
if _CONNECTION_SUPPORTS_TIMEOUT:
connection = connection_class(host, timeout=deadline)
else:
connection = connection_class(host)
(lines 376-379 in my case) and replace it with:
if _CONNECTION_SUPPORTS_TIMEOUT:
if host[:9] == 'localhost' or host[:9] == '127.0.0.1':
connection = connection_class(host, timeout=deadline)
else:
connection = connection_class('your_proxy_host_goes_here', your_proxy_port_number_goes_here, timeout=deadline)
else:
if host[:9] == 'localhost' or host[:9] == '127.0.0.1':
connection = connection_class(host)
else:
connection = connection_class('your_proxy_host_goes_here', your_proxy_port_number_goes_here)
replacing the placeholders your_proxy_host_goes_here and your_proxy_port_number_goes_here with appropriate values.
(I believe this code can be written more elegantly, though... any Python geeks out there? :) )
In my case, I also had to delete the existing compiled file urlfetch_stub.pyc (located in the same directory as urlfetch_stub.py) because the SDK didn't seem to pick up the changes until I did so.
Now you can use dev_appserver to launch your app, and use urlfetch-backed services within the app, free from HTTP 403 errors.

Tweepy SSLError

I have a Django management command, launched via supervisord, that uses tweepy to consume the twitter streaming API.
The agent works quite well however I notice in the logs there's an SSLError every 10-15 minutes and supervisord is re-launching the agent.
The tweepy package is latest, version 1.11. The server is ubuntu 12.04 LTS. I've tried installing the cacert into the key chain as mentioned in the link below, but no luck.
Twitter API SSL Root CA Certificate
Any suggestions?
[2012-08-26 19:28:15,656: ERROR] Error establishing the connection
Traceback (most recent call last):.../.../datasinks.py", line 102, in start
stream.filter(locations=self.locations)
File "/site/pythonenv/local/lib/python2.7/site-packages/tweepy/streaming.py", line 228, in filter
self._start(async)
File "/site/pythonenv/local/lib/python2.7/site-packages/tweepy/streaming.py", line 172, in _start
self._run()
File "/site/pythonenv/local/lib/python2.7/site-packages/tweepy/streaming.py", line 117, in _run
self._read_loop(resp)
File "/site/pythonenv/local/lib/python2.7/site-packages/tweepy/streaming.py", line 150, in _read_loop
c = resp.read(1)
File "/usr/lib/python2.7/httplib.py", line 541, in read
return self._read_chunked(amt)
File "/usr/lib/python2.7/httplib.py", line 574, in _read_chunked
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib/python2.7/ssl.py", line 241, in recv
return self.read(buflen)
File "/usr/lib/python2.7/ssl.py", line 160, in read
return self._sslobj.read(len)
SSLError: The read operation timed out
Following is an outline of the code.
from tweepy import API, OAuthHandler
from tweepy.streaming import StreamListener, Stream
# snip other imports
class TwitterSink(StreamListener, TweetSink):
def __init__(self):
self.auth = OAuthHandler(settings.TWITTER_OAUTH_CONSUMER_KEY, settings.TWITTER_OAUTH_CONSUMER_SECRET)
self.auth.set_access_token(settings.TWITTER_OAUTH_ACCESS_TOKEN_KEY, settings.TWITTER_OAUTH_ACCESS_TOKEN_SECRET)
self.locations = '' # Snip for brevity
def start(self):
try:
stream = Stream(self.auth, self,timeout=60, secure=True)
stream.filter(locations=self.locations)
except SSLError as e:
logger.exception("Error establishing the connection")
except IncompleteRead as r:
logger.exception("Error with HTTP connection")
# snip on_data()
# snip on_timeout()
# snip on_error()

The certificate doesn't seem to be the problem. The error is just a timeout. Seems like an issue with tweepy's SSL handling to me. The code is equipped to handle socket.timeout and reopen the connection, but not a timeout arriving through SSLError.
Looking at the ssl module code (or docs), though, I don't see a pretty way to catch that. The SSLError object is raised without any arguments, just a string description. For lack of a better solution, I'd suggest adding the following right before line 118 of tweepy/streaming.py:
except SSLError, e:
if 'timeout' not in exception.message.lower(): # support all timeouts
exception = e
break
if self.listener.on_timeout() == False:
break
if self.running is False:
break
conn.close()
sleep(self.snooze_time)
Why it's timing out in the first place is a good question. I have nothing better than repeating Travis Mehlinger's suggestion of setting a higher timeout.

Here is how I have it (modified solution from here https://groups.google.com/forum/?fromgroups=#!topic/tweepy/80Ayu1joGJ4):
l = MyListener()
auth = OAuthHandler(settings.CONSUMER_KEY, settings.CONSUMER_SECRET)
auth.set_access_token(settings.ACCESS_TOKEN, settings.ACCESS_TOKEN_SECRET)
# connect to stream
stream = Stream(auth, l, timeout=30.0)
while True:
# Call tweepy's userstream method with async=False to prevent
# creation of another thread.
try:
stream.filter(follow=reporters, async=False)
# Normal exit: end the thread
break
except Exception, e:
# Abnormal exit: Reconnect
logger.error(e)
nsecs = random.randint(60, 63)
logger.error('{0}: reconnect in {1} seconds.'.format(
datetime.datetime.utcnow(), nsecs))
time.sleep(nsecs)

There is another alternative solution provided on Github:
https://github.com/tweepy/tweepy/pull/132

App Engine urlfetch is raising exceptions when i think it should not be

I've written my first Python application with the App Engine APIs, it is intended to monitor a list of servers and notify me when one of them goes down, by sending a message to my iPhone using Prowl, or sending me an email, or both.
Problem is, a few times a week it notifies me a server is down even when it clearly isn't. I've tested it with servers i know should be up virtually all the time like google.com or amazon.com but i get notifications with them too.
I've got a copy of the code running at http://aeservmon.appspot.com, you can see that google.com was added Jan 3rd but is only listed as being up for 6 days.
Below is the relevant section of the code from checkservers.py that does the checking with urlfetch, i assumed that the DownloadError exception would only be raised when the server couldn't be contacted, but perhaps I'm wrong.
What am I missing?
Full source on github under mrsteveman1/aeservmon (i can only post one link as a new user, sorry!)
def testserver(self,server):
if server.ssl:
prefix = "https://"
else:
prefix = "http://"
try:
url = prefix + "%s" % server.serverdomain
result = urlfetch.fetch(url, headers = {'Cache-Control' : 'max-age=30'} )
except DownloadError:
logging.info('%s could not be reached' % server.serverdomain)
self.serverisdown(server,000)
return
if result.status_code == 500:
logging.info('%s returned 500' % server.serverdomain)
self.serverisdown(server,result.status_code)
else:
logging.info('%s is up, status code %s' % (server.serverdomain,result.status_code))
self.serverisup(server,result.status_code)
UPDATE Jan 21:
Today I found one of the exceptions in the logs:
ApplicationError: 5
Traceback (most recent call last):
File "/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 507, in __call__
handler.get(*groups)
File "/base/data/home/apps/aeservmon/1.339312180538855414/checkservers.py", line 149, in get
self.testserver(server)
File "/base/data/home/apps/aeservmon/1.339312180538855414/checkservers.py", line 106, in testserver
result = urlfetch.fetch(url, headers = {'Cache-Control' : 'max-age=30'} )
File "/base/python_lib/versions/1/google/appengine/api/urlfetch.py", line 241, in fetch
return rpc.get_result()
File "/base/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 501, in get_result
return self.__get_result_hook(self)
File "/base/python_lib/versions/1/google/appengine/api/urlfetch.py", line 331, in _get_fetch_result
raise DownloadError(str(err))
DownloadError: ApplicationError: 5

other folks have been reporting issues with the fetch service (e.g. http://code.google.com/p/googleappengine/issues/detail?id=1902&q=urlfetch&colspec=ID%20Type%20Status%20Priority%20Stars%20Owner%20Summary%20Log%20Component)
can you print the exception, it may have more detail, e.g.:
"DownloadError: ApplicationError: 2 something bad"