Develop Reference

Issue during import, MLRunNotFoundError

I installed python package MLRun correctly, but I got in jupyter this error
---------------------------------------------------------------------------
HTTPError Traceback (most recent call last)
/opt/conda/lib/python3.8/site-packages/mlrun/errors.py in raise_for_status(response, message)
75 try:
---> 76 response.raise_for_status()
77 except requests.HTTPError as exc:
/opt/conda/lib/python3.8/site-packages/requests/models.py in raise_for_status(self)
942 if http_error_msg:
--> 943 raise HTTPError(http_error_msg, response=self)
944
HTTPError: 404 Client Error: Not Found for url: https://mlrun-api.default-tenant.app.iguazio.prod//api/v1/client-spec
...
/opt/conda/lib/python3.8/site-packages/mlrun/errors.py in raise_for_status(response, message)
80 error_message = f"{str(exc)}: {message}"
81 try:
---> 82 raise STATUS_ERRORS[response.status_code](
83 error_message, response=response
84 ) from exc
MLRunNotFoundError: 404 Client Error: Not Found for url: https://mlrun-api.default-tenant.app.iguazio.prod//api/v1/client-spec: details: Not Found
based on this source code in python:
%pip install mlrun scikit-learn~=1.0
import mlrun
...
Do you see issues?
BTW: I installed relevant MLRun version (the same as on server side). I modified file mlrun.env and added values for these variables MLRUN_DBPATH, V3IO_USERNAME, V3IO_ACCESS_KEY

Issue
I got it, the problem is in invalid url, see error message Not Found for url: https://mlrun-api.default-tenant.app.iguazio.prod//api/v1/client-spec.
You can see double // and it is the issue, variable MLRUN_DB contains this url 'https://mlrun-api.default-tenant.app.iguazio.prod/'.
Solution
You have to remove end / in MLRUN_DB and final value will be MLRUN_DB=https://mlrun-api.default-tenant.app.iguazio.prod
Now it works.

"Unhandled auth type: app, user" from dropbox api

Until the other day I was able to read files using the dropbox API, but now I get an authentication error.
!pip install dropbox
import dropbox
import time
dbx = dropbox.Dropbox(<Access Token>)
dir = <folder_path>
lists = []
def __get_files_recursive(res):
for entry in res.entries:
##print(list(entry.path_display.split(",")))
lists.append(entry.path_display.split(","))
##print(lists)
print(entry.path_display)
if (res.has_more):
res2 = dbx.files_list_folder_continue(res.cursor)
__get_files_recursive(res2)
res = dbx.files_list_folder(dir, recursive=True)
__get_files_recursive(res)
Error Code was as follows
---------------------------------------------------------------------------
BadInputException Traceback (most recent call last)
<ipython-input-64-f64dcbce5290> in <module>()
12 __get_files_recursive(res2)
13
---> 14 res = dbx.files_list_folder(dir, recursive=True)
15 __get_files_recursive(res)
3 frames
/usr/local/lib/python3.7/dist-packages/dropbox/dropbox_client.py in request_json_string(self, host, func_name, route_style, request_json_arg, auth_type, request_binary, timeout)
562 pass
563 else:
--> 564 raise BadInputException('Unhandled auth type: {}'.format(auth_type))
565
566 # The contents of the body of the HTTP request
BadInputException: Unhandled auth type: app, user
If you know how to solve this problem, please let me know.

invalid literal for int() with base 16: b''

Background - I am trying to create a Twitter bot which listens in real time for certain keywords coming only from certain accounts. When these conditions are met, I would then retweet it with my app, so I can follow with notifications from my main account.
Problem - Sometimes, but not always, I will encounter an error while the script is running which gives me the error listed in the subject of this post. I'm unsure what causes it, since it happens intermittently.
What I've Done - I've searched for the error; most of what I've found refers to "base 10", not "base 16". For the couple cases I've found where it references base 16, I don't understand the solution well enough to adapt it to my code (self teaching myself on this project).
Code
import tweepy
import json
import re
import logging
auth = tweepy.OAuthHandler("xxxx", "xxxx")
auth.set_access_token("yyyyy", "yyyy")
api = tweepy.API(auth, wait_on_rate_limit=True,
wait_on_rate_limit_notify=True)
class MyStreamListener(tweepy.StreamListener):
def __init__(self, api):
self.api = api
self.me = api.me()
def on_status(self, tweet):
keyword = ["Keyword1", "Keyword2"]
accounts = ['account1','account2']
patterns = [r'\b%s\b' % re.escape(s.strip()) for s in keyword]
patterns2 = [r'\b%s\b' % re.escape(s.strip()) for s in accounts]
there = re.compile('|'.join(patterns))
there2 = re.compile('|'.join(patterns2))
if there.search(tweet.text) and there2.search(str(tweet.user.id)):
print("New")
tweet.retweet()
def on_error(self, status):
print("Error detected")
tweets_listener = MyStreamListener(api)
stream = tweepy.Stream(api.auth, tweets_listener)
stream.filter(follow=['account1','account2'])
Error
ValueError Traceback (most recent call last)
~\anaconda3\lib\http\client.py in _get_chunk_left(self)
554 try:
555 chunk_left = self._read_next_chunk_size()
556 except ValueError:
~\anaconda3\lib\http\client.py in _read_next_chunk_size(self)
521 try:
522 return int(line, 16)
523 except ValueError:
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
IncompleteRead Traceback (most recent call last)
I then get a couple of incomplete reads, the last line of the error is:
ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

This should now be handled in the development version of Tweepy on the master branch by automatically attempting to reconnect. The underlying connection error is likely due to falling too far behind in the stream when processing Tweets. For now, you can attempt to reduce the processing within on_status, handle the error by restarting the stream, wait for Tweepy v4.0, or use the development version of Tweepy.
For more context, see https://github.com/tweepy/tweepy/issues/237 and https://github.com/tweepy/tweepy/issues/448.

Docker CLI allows tag, but Docker Python API raises APIError

I am trying to push a local Docker image to ECR using the Docker Python API. As part of that process I need to tag the image a certain way. When I do so on the CLI, it works:
docker tag foo/bar '{user_id}.dkr.ecr.us-east-1.amazonaws.com/foo/bar'
However when I try to do the same thing using the docker.images.Image.tag function in the Docker Python SDK it fails:
import docker
(docker.client.from_env().images.get('foo/bar')
.tag('foo/bar',
'{user-id}.dkr.ecr.us-east-1.amazonaws.com/foo/bar'
)
)
(replace user_id in the code samples above with an AWS user id value, e.g. 717171717171; I've obfuscated it here for the purposes of this question)
With the following error:
In [10]: docker.client.from_env().images.get('foo/bar').ta
...: g('foo/bar', '{user_id}.dkr.ecr.us-east-1.amaz
...: onaws.com/foo/bar')
---------------------------------------------------------------------------
HTTPError Traceback (most recent call last)
~/miniconda3/envs/alekseylearn-dev/lib/python3.6/site-packages/docker/api/client.py in _raise_for_status(self, response)
255 try:
--> 256 response.raise_for_status()
257 except requests.exceptions.HTTPError as e:
~/miniconda3/envs/alekseylearn-dev/lib/python3.6/site-packages/requests/models.py in raise_for_status(self)
939 if http_error_msg:
--> 940 raise HTTPError(http_error_msg, response=self)
941
HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/images/sha256:afe07035bce72b6c496878a7e3960bedffd46c1bedc79f1bd2b89619e8457194/tag?tag={user_id}.dkr.ecr.us-east-1.amazonaws.com%2Ffoo%2Fbar&repo=foo%2Fbar&force=0
During handling of the above exception, another exception occurred:
APIError Traceback (most recent call last)
<ipython-input-10-5bb015d17409> in <module>
----> 1 docker.client.from_env().images.get('alekseylearn-example/build').tag('foo/bar', '{user_id}.dkr.ecr.us-east-1.amazonaws.com/foo/bar')
~/miniconda3/envs/alekseylearn-dev/lib/python3.6/site-packages/docker/models/images.py in tag(self, repository, tag, **kwargs)
120 (bool): ``True`` if successful
121 """
--> 122 return self.client.api.tag(self.id, repository, tag=tag, **kwargs)
123
124
~/miniconda3/envs/alekseylearn-dev/lib/python3.6/site-packages/docker/utils/decorators.py in wrapped(self, resource_id, *args, **kwargs)
17 'Resource ID was not provided'
18 )
---> 19 return f(self, resource_id, *args, **kwargs)
20 return wrapped
21 return decorator
~/miniconda3/envs/alekseylearn-dev/lib/python3.6/site-packages/docker/api/image.py in tag(self, image, repository, tag, force)
531 url = self._url("/images/{0}/tag", image)
532 res = self._post(url, params=params)
--> 533 self._raise_for_status(res)
534 return res.status_code == 201
535
~/miniconda3/envs/alekseylearn-dev/lib/python3.6/site-packages/docker/api/client.py in _raise_for_status(self, response)
256 response.raise_for_status()
257 except requests.exceptions.HTTPError as e:
--> 258 raise create_api_error_from_http_exception(e)
259
260 def _result(self, response, json=False, binary=False):
~/miniconda3/envs/alekseylearn-dev/lib/python3.6/site-packages/docker/errors.py in create_api_error_from_http_exception(e)
29 else:
30 cls = NotFound
---> 31 raise cls(e, response=response, explanation=explanation)
32
33
APIError: 500 Server Error: Internal Server Error ("invalid tag format")
Why does the CLI command succeed and the Python API command fail?

In detailed Docker API lingo, an image name like 123456789012.dkr.ecr.us-east-1.amazon.aws.com/foo/bar:baz is split up into a repository (before the colon) and a tag (after the colon). The host-name part of the repository name is a registry. The default tag value if none is specified is the literal latest.
In your case, you already have an Image object, so you need to apply the two "halves" of the second argument:
docker.client.from_env().images.get('foo/bar')
.tag('{user-id}.dkr.ecr.us-east-1.amazonaws.com/foo/bar',
'latest'
)
(In many practical cases using the latest tag isn't a great idea; something like a timestamp or source control commit ID better identifies the image and helps indicate to services like ECS or EKS or plain Kubernetes that they need to do an update. Also, while the ECR image IDs are kind of impractically long, in a scripting context nothing stops you from using them directly; you can, for example, docker build -t 12345...amazonaws.com/foo/bar:abcdef0 and skip the intermediate docker tag step if you want.)

python-twitter: get_oauth( ) giving unexpected error

I am new to python and I have been making codes to scrap twitter data on python.
Below are my codes:
import csv
import json
import twitter_oauth
import sys
sys.path.append("/Users/jdschnieder/Documents/Modules")
print sys.path
#gain authorization to twitter
consumer_key = 'xdbX1g21REs0MPxofJPcFw'
consumer_secret = 'c9S9YI6G3GIdjkg67ecBaCXSJDlGw4sybTv1ccnkrA'
get_oauth_obj = twitter_oauth.GetOauth(consumer_key, consumer_secret)
get_oauth_obj.get_oauth()
the error occurs at the line:
get_oauth_obj.get_oauth()
the error message is as follows:
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-36-5d124f99deb6> in <module>()
--> 1 get_oauth_obj.get_oauth()
/Users/jdschnieder/anaconda/python.app/Contents/lib/python2.7/site-packages/
twitter_oauth-0.2.0-py2.7.egg/twitter_oauth.pyc in get_oauth(self)
95 resp, content = client.request(_REQUEST_TOKEN_URL, "GET")
96 if resp['status'] != '200':
-> 97 raise Exception('Invalid response %s' % resp['status'])
98
99 request_token = dict(self._parse_qsl(content))
Exception: Invalid response 401
why is this error occurring, and what are possible solutions to the error?
thank you,

It looks like the Twitter library you're using does not support Twitter API v1.1. Instead of twitter_oauth, use one of the Python libraries listed here. For example, you can use Tweepy, following the documentation for OAuth Authentication