I have been struggling with this problem for a while now and have scoured the internet but am still unable to solve it.
I am trying to run a python script to access a google sheet from my local machine at work (so may be firewall issues). Whenever I am connected on a guest network the script works fine. Whenever I try to use my work internal network (with or without proxy settings) I get the following error:
***Traceback (most recent call last):
File "C:\Python36-32\lib\site-packages\httplib2\__init__.py", line 995, in _conn_request
conn.connect()
File "C:\Python36-32\lib\http\client.py", line 1392, in connect
super().connect()
File "C:\Python36-32\lib\http\client.py", line 936, in connect
(self.host,self.port), self.timeout, self.source_address)
File "C:\Python36-32\lib\socket.py", line 704, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "C:\Python36-32\lib\socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11004] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.1\helpers\pydev\pydevd.py", line 1664, in <module>
main()
File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.1\helpers\pydev\pydevd.py", line 1658, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.1\helpers\pydev\pydevd.py", line 1068, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.1\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Git/datalab-notebooks/2017-vin-profitability/US/Finance_Mapping_test.py", line 23, in <module>
client = gspread.authorize(credentials)
File "C:\Python36-32\lib\site-packages\gspread\__init__.py", line 38, in authorize
client.login()
File "C:\Python36-32\lib\site-packages\gspread\client.py", line 52, in login
self.auth.refresh(http)
File "C:\Python36-32\lib\site-packages\oauth2client\client.py", line 668, in refresh
self._refresh(http.request)
File "C:\Python36-32\lib\site-packages\oauth2client\client.py", line 873, in _refresh
self._do_refresh_request(http_request)
File "C:\Python36-32\lib\site-packages\oauth2client\client.py", line 905, in _do_refresh_request
self.token_uri, method='POST', body=body, headers=headers)
File "C:\Python36-32\lib\site-packages\httplib2\__init__.py", line 1322, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\Python36-32\lib\site-packages\httplib2\__init__.py", line 1072, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\Python36-32\lib\site-packages\httplib2\__init__.py", line 1002, in _conn_request
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
httplib2.ServerNotFoundError: Unable to find the server at accounts.google.com***
After some research I get the impression that it is due to my proxy settings but I cant figure out what I need to do to fix it. I also understand that the python library socks may help but once again I'm stumped
Any help would be much appreciated
Thanks in advance
While this was asked a long time ago, I wanted to share the fix that worked for me.
The error occurred when I used the host with the protocol
httplib2.ProxyInfo(
proxy_type=httplib2.socks.PROXY_TYPE_HTTP,
proxy_host='https://proxy.prod.com',
)
Removing the protocol solved the problem:
httplib2.ProxyInfo(
proxy_type=httplib2.socks.PROXY_TYPE_HTTP,
proxy_host='proxy.prod.com',
)
Related
My goal is to read file from hdfs in airflow and do further manipulations.
After researching, I found that url I need to use is as follows:
df = pd.read_parquet('http://localhost:9870/webhdfs/v1/hadoop_files/sample_2022_01.parquet?op=OPEN'),
where localhost/172.20.80.1/computer-name.mshome.net can be interchangeably used,
9870 - namenode port,
hadoop_files/sample_2022_01.parquet - my folder and file created in the root.
I can access and read file locally in PyCharm, but I am unable to get the same result inside airflow in docker. I tried using local hdfs and hdfs hosted in docker and changing host to the host.docker.internal, but I am getting the same error.
Stack trace:
[2022-06-12, 17:52:45 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.7/urllib/request.py", line 1350, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/local/lib/python3.7/http/client.py", line 976, in send
self.connect()
File "/usr/local/lib/python3.7/http/client.py", line 948, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
OSError: [Errno 113] No route to host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 207, in execute
branch = super().execute(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 171, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 189, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/includes/parquet_dag/main.py", line 15, in main
df_parquet = read('hdfs://localhost:9000/hadoop_files/sample_2022_01.parquet')
File "/opt/airflow/dags/includes/parquet_dag/utils.py", line 29, in read
df = pd.read_parquet('http://172.20.80.1:9870/webhdfs/v1/hadoop_files/sample_2022_01.parquet?op=OPEN')
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 500, in read_parquet
**kwargs,
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 236, in read
mode="rb",
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 102, in _get_path_or_handle
path_or_handle, mode, is_text=False, storage_options=storage_options
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/common.py", line 614, in get_handle
storage_options=storage_options,
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/common.py", line 312, in _get_filepath_or_buffer
with urlopen(req_info) as req:
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/common.py", line 212, in urlopen
return urllib.request.urlopen(*args, **kwargs)
File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/local/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.7/urllib/request.py", line 1378, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/lib/python3.7/urllib/request.py", line 1352, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 113] No route to host>
With host.docker.internal:
urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address>
you need to use any routable address inside airflow docker container.
if hadoop is inside docker container as well, check it ip address using docker inspect CONTAINER (doc). if hadoop is on localhost you can set network_mode: "host" (doc)
also there is an important notice if you are on macos and have the docker desktop app which basically a virtual machine. so in this case you need some extra settings, check this, for example.
where localhost/172.20.80.1/computer-name.mshome.net can be interchangeably used,
They shouldn't be interchangeable inside Docker network.
From Airflow, you could use Docker service names, not IP addresses, and ensure the containers are in the same bridge network (not host mode, which only works on Linux). host.docker.internal isn't correct either since you're trying to reach another container, not your host
https://docs.docker.com/network/bridge/
I'd also recommend using Airflow Spark operators to actually read Parquet from HDFS, using Spark, not Pandas or WebHDFS. You can convert Spark dataframes to Pandas, if needed
I'm trying to use pygsheets in a script on ScrapingHub. The pygsheets part of the script begins with:
google_client = pygsheets.authorize(service_file=CREDENTIALS_FILENAME, no_cache=True)
spreadsheet = google_client.open_by_key(SHEET_ID)
Where CREDENTIALS_FILENAME is my Google service account credentials file name and SHEET_ID is the spreadsheet ID; at some point of that, this is happening:
2018-01-30 16:27:36 INFO [stdout] service_email : *******#**********.iam.gserviceaccount.com
2018-01-30 16:27:36 INFO [googleapiclient.discovery] URL being requested: GET https://www.googleapis.com/drive/v3/files?corpora=user&pageSize=500&fields=files%28id%2C+name%29&q=mimeType%3D%27application%2Fvnd.google-apps.spreadsheet%27&supportsTeamDrives=false&includeTeamDriveItems=false&alt=json
2018-01-30 16:27:36 INFO [oauth2client.transport] Attempting refresh to obtain initial access_token
2018-01-30 16:27:36 INFO [oauth2client.client] Refreshing access_token
2018-01-30 16:31:50 ERROR [root] Job runtime exception
Traceback (most recent call last):
File "/app/python/lib/python3.6/site-packages/httplib2/__init__.py", line 995, in _conn_request
conn.connect()
File "/usr/local/lib/python3.6/http/client.py", line 1392, in connect
super().connect()
File "/usr/local/lib/python3.6/http/client.py", line 936, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/lib/python3.6/socket.py", line 704, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 148, in _run_usercode
_run(args, settings)
File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 105, in _run
_run_pkgscript(args)
File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 128, in _run_pkgscript
d.run_script(scriptname, {'__name__': '__main__'})
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1517, in run_script
exec(code, namespace, namespace)
File "/tmp/unpacked-eggs/__main__.egg/EGG-INFO/scripts/update_gsheets.py", line 190, in <module>
main(parse_args())
File "/tmp/unpacked-eggs/__main__.egg/EGG-INFO/scripts/update_gsheets.py", line 152, in main
google_client = pygsheets.authorize(service_file=CREDENTIALS_FILENAME, no_cache=True)
File "/app/python/lib/python3.6/site-packages/pygsheets/client.py", line 552, in authorize
rclient = Client(oauth=credentials, **client_kwargs)
File "/app/python/lib/python3.6/site-packages/pygsheets/client.py", line 79, in __init__
self._fetch_sheets()
File "/app/python/lib/python3.6/site-packages/pygsheets/client.py", line 92, in _fetch_sheets
results = self._execute_request(None, request, False)
File "/app/python/lib/python3.6/site-packages/pygsheets/client.py", line 418, in _execute_request
response = request.execute()
File "/app/python/lib/python3.6/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/app/python/lib/python3.6/site-packages/googleapiclient/http.py", line 839, in execute
method=str(self.method), body=self.body, headers=self.headers)
File "/app/python/lib/python3.6/site-packages/googleapiclient/http.py", line 166, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/app/python/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
redirections, connection_type)
File "/app/python/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
connection_type=connection_type)
File "/app/python/lib/python3.6/site-packages/httplib2/__init__.py", line 1322, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/app/python/lib/python3.6/site-packages/httplib2/__init__.py", line 1072, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/app/python/lib/python3.6/site-packages/httplib2/__init__.py", line 1002, in _conn_request
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com
I've tested the script before and it used to work, so I don't know what is happening. What is this happening? There is some configuration or authentication step that I missed?
I just started with Nagios XI and i got this error and i cant find the right answer does anyone know what the problem is?
I already checked the proxy and i cant find where the problem is.
The code is to get notifications messages through whats app.
[root#localhost ~]# yowsup-cli registration --requestcode sms --phone 316******** --cc 31 --mcc 204 --mnc 10
yowsup-cli v2.0.15
yowsup v2.5.0
Copyright (c) 2012-2016 Tarek Galal
http://www.openwhatsapp.org
This software is provided free of charge. Copying and redistribution is
encouraged.
If you appreciate this software and you would like to support future
development please consider donating:
http://openwhatsapp.org/yowsup/donate
Traceback (most recent call last):
File "/usr/bin/yowsup-cli", line 368, in <module>
if not parser.process():
File "/usr/bin/yowsup-cli", line 189, in process
self.handleRequestCode(self.args["requestcode"], config)
File "/usr/bin/yowsup-cli", line 208, in handleRequestCode
result = codeReq.send()
File "/usr/lib/python2.6/site-packages/yowsup/registration/coderequest.py", line 62, in send
res = super(WACodeRequest, self).send(parser)
File "/usr/lib/python2.6/site-packages/yowsup/common/http/warequest.py", line 70, in send
return self.sendGetRequest(parser)
File "/usr/lib/python2.6/site-packages/yowsup/common/http/warequest.py", line 108, in sendGetRequest
self.response = WARequest.sendRequest(host, port, path, headers, params, "GET")
File "/usr/lib/python2.6/site-packages/yowsup/common/http/warequest.py", line 164, in sendRequest
conn.request(reqType, path, params, headers);
File "/usr/lib64/python2.6/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.6/httplib.py", line 1010, in _send_request
self.endheaders()
File "/usr/lib64/python2.6/httplib.py", line 967, in endheaders
self._send_output()
File "/usr/lib64/python2.6/httplib.py", line 831, in _send_output
self.send(msg)
File "/usr/lib64/python2.6/httplib.py", line 790, in send
self.connect()
File "/usr/lib64/python2.6/httplib.py", line 1171, in connect
sock = socket.create_connection((self.host, self.port), self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
socket.error: [Errno 111] Connection refused
thanks in advance.
I'm trying to get a Flask/Neo4j app set up on a remote Ubuntu server, and I've run into a problem that I haven't been able to figure out. My app uses py2neo, but when it tries to connect to the graph, the app crashes and the Neo4j process seems to stop. I've tried connecting in a python shell like this...
test = Graph('http://localhost:7474/db/data/',username='neo4j',password='myPassword')
which fails, and also renders neo4j inoperative until I restart it. However, these return 200 responses (and the web interface also works):
curl -u neo4j http://localhost:7474/db/data/
requests.get('http://localhost:7474/db/data/', auth=('neo4j','myPassword'))
I've tried to provide more information than this similar question, because it seems like the connection works from everywhere but py2neo.
Here's the full traceback:
Traceback (most recent call last):
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/database/__init__.py", line 318, in __new__
inst = cls.__instances[key]
KeyError: (<class 'py2neo.database.Graph'>, <ServerAddress settings={'http_port': 7474, 'host': 'localhost'}>, 'data')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 322, in submit
response = send()
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 317, in send
http.request(xstr(method), xstr(uri.absolute_path_reference), body, headers)
File "/usr/lib/python3.5/http/client.py", line 1106, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/usr/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 80, in connect
self.source_address)
File "/usr/lib/python3.5/socket.py", line 711, in create_connection
raise err
File "/usr/lib/python3.5/socket.py", line 702, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/database/__init__.py", line 327, in __new__
use_bolt = version_tuple(inst.__remote__.get().content["neo4j_version"]) >= (3,)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/database/http.py", line 154, in get
response = self.__base.get(headers=headers, redirect_limit=redirect_limit, **kwargs)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 966, in get
return self.__get_or_head("GET", if_modified_since, headers, redirect_limit, **kwargs)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 943, in __get_or_head
return rq.submit(redirect_limit=redirect_limit, **kwargs)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 433, in submit
http, rs = submit(self.method, uri, self.body, self.headers)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 362, in submit
raise SocketError(code, description, host_port=uri.host_port)
py2neo.packages.httpstream.http.SocketError: Connection refused
Anything I might try to figure out what's going on would be appreciated.
Changed to http://username:password#localhost:7474/db/data/ and it works!
Example:
test = Graph('http://username:password#localhost:7474/db/data/')
I had the same issue, solved with a simple upgrade of pip version.
pip install --upgrade py2neo
I am new to Python and trying to run attached code.
import urllib
url = 'https://www.google.com/'
print(url)
sock = urllib.urlopen(url)
But I get an error below:
>>> runfile('C:/Users/myname/Documents/Python Scripts/ex00.py', wdir=r'C:/Users/myname/Documents/Python Scripts')
https://www.google.com/
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\myname\AppData\Local\Continuum\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 540, in runfile
execfile(filename, namespace)
File "C:/Users/myname/Documents/Python Scripts/ex00.py", line 5, in <module>
sock = urllib.urlopen(url)
File "C:\Users\myname\AppData\Local\Continuum\Anaconda\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Users\myname\AppData\Local\Continuum\Anaconda\lib\urllib.py", line 208, in open
return getattr(self, name)(url)
File "C:\Users\myname\AppData\Local\Continuum\Anaconda\lib\urllib.py", line 437, in open_https
h.endheaders(data)
File "C:\Users\myname\AppData\Local\Continuum\Anaconda\lib\httplib.py", line 969, in endheaders
self._send_output(message_body)
File "C:\Users\myname\AppData\Local\Continuum\Anaconda\lib\httplib.py", line 829, in _send_output
self.send(msg)
File "C:\Users\myname\AppData\Local\Continuum\Anaconda\lib\httplib.py", line 791, in send
self.connect()
File "C:\Users\myname\AppData\Local\Continuum\Anaconda\lib\httplib.py", line 1172, in connect
self.timeout, self.source_address)
File "C:\Users\myname\AppData\Local\Continuum\Anaconda\lib\socket.py", line 571, in create_connection
raise err
IOError: [Errno socket error] [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Earlier I thought that it is may be because I am running the script from my office and VPN is causing issues. So I ran it from my home internet connection but no resolution :(