I'm working on a project that is written in Python and needs to post updates to a Friendica server and interact using various APIs available. However, I have had very limited API usage experience and so I'm unsure of how to code this functionality in Python. There is an example on the Friendica GitHub however the Python example would not work for me. There is a Python3 module https://bitbucket.org/tobiasd/python-friendica/overview, however when trying to connect using this in a test script as follows:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import friendica
# make a new instance of friendica
f = friendica.friendica (server = '10.211.55.23/api/statuses/update', username = 'newtest', password = 'klaup8744')
# check that we are logged in
f.account_verify_credentials()
# get the current notifications
print (f.ping())
# post something with the default settings
f.statuses_update( status = "here is the message you are going to post" )
it would refuse the connection with the following message:
Traceback (most recent call last):
File "/usr/lib/python3.4/urllib/request.py", line 1182, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/usr/lib/python3.4/http/client.py", line 1088, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.4/http/client.py", line 1126, in _send_request
self.endheaders(body)
File "/usr/lib/python3.4/http/client.py", line 1084, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.4/http/client.py", line 922, in _send_output
self.send(msg)
File "/usr/lib/python3.4/http/client.py", line 857, in send
self.connect()
File "/usr/lib/python3.4/http/client.py", line 1223, in connect
super().connect()
File "/usr/lib/python3.4/http/client.py", line 834, in connect
self.timeout, self.source_address)
File "/usr/lib/python3.4/socket.py", line 512, in create_connection
raise err
File "/usr/lib/python3.4/socket.py", line 503, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test_1.py", line 12, in <module>
print (f.ping())
File "/home/sambraidley/Desktop/friendica.py", line 851, in ping
res = urlopen(self.protocol()+self.apipath[:-4]+'/ping').read().decode('utf-8')
File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 463, in open
response = self._open(req, data)
File "/usr/lib/python3.4/urllib/request.py", line 481, in _open
'_open', req)
File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 1225, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/usr/lib/python3.4/urllib/request.py", line 1184, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
The friendica Python3 module can be found at the following https://bitbucket.org/tobiasd/python-friendica/src/b0b75ae80a6e747e8724b1ae36972ebfd939beb5/friendica.py?fileviewer=file-view-default
My Friendica server is setup within a VM with the address 10.211.55.23, with test credentials of username = 'newtest' and password 'klaup8744' and it is fully working as using the curl example code to post an update worked perfectly, as follows:
/usr/bin/curl -u newtest:klaup8744 10.211.55.23/api/statuses/update.xml -d source="Testing" -d status="This is a test status"
According to the source link you posted, friendica uses https by default. Your successful curl request is using http.
Try instantiating friendica it using http:
f = friendica.friendica (server = '10.211.55.23/api/statuses/update',
username = 'newtest', password = 'klaup8744', useHTTPS=False)
Related
My goal is to read file from hdfs in airflow and do further manipulations.
After researching, I found that url I need to use is as follows:
df = pd.read_parquet('http://localhost:9870/webhdfs/v1/hadoop_files/sample_2022_01.parquet?op=OPEN'),
where localhost/172.20.80.1/computer-name.mshome.net can be interchangeably used,
9870 - namenode port,
hadoop_files/sample_2022_01.parquet - my folder and file created in the root.
I can access and read file locally in PyCharm, but I am unable to get the same result inside airflow in docker. I tried using local hdfs and hdfs hosted in docker and changing host to the host.docker.internal, but I am getting the same error.
Stack trace:
[2022-06-12, 17:52:45 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.7/urllib/request.py", line 1350, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/local/lib/python3.7/http/client.py", line 976, in send
self.connect()
File "/usr/local/lib/python3.7/http/client.py", line 948, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
OSError: [Errno 113] No route to host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 207, in execute
branch = super().execute(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 171, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 189, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/includes/parquet_dag/main.py", line 15, in main
df_parquet = read('hdfs://localhost:9000/hadoop_files/sample_2022_01.parquet')
File "/opt/airflow/dags/includes/parquet_dag/utils.py", line 29, in read
df = pd.read_parquet('http://172.20.80.1:9870/webhdfs/v1/hadoop_files/sample_2022_01.parquet?op=OPEN')
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 500, in read_parquet
**kwargs,
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 236, in read
mode="rb",
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/parquet.py", line 102, in _get_path_or_handle
path_or_handle, mode, is_text=False, storage_options=storage_options
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/common.py", line 614, in get_handle
storage_options=storage_options,
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/common.py", line 312, in _get_filepath_or_buffer
with urlopen(req_info) as req:
File "/home/airflow/.local/lib/python3.7/site-packages/pandas/io/common.py", line 212, in urlopen
return urllib.request.urlopen(*args, **kwargs)
File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/local/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.7/urllib/request.py", line 1378, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/lib/python3.7/urllib/request.py", line 1352, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 113] No route to host>
With host.docker.internal:
urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address>
you need to use any routable address inside airflow docker container.
if hadoop is inside docker container as well, check it ip address using docker inspect CONTAINER (doc). if hadoop is on localhost you can set network_mode: "host" (doc)
also there is an important notice if you are on macos and have the docker desktop app which basically a virtual machine. so in this case you need some extra settings, check this, for example.
where localhost/172.20.80.1/computer-name.mshome.net can be interchangeably used,
They shouldn't be interchangeable inside Docker network.
From Airflow, you could use Docker service names, not IP addresses, and ensure the containers are in the same bridge network (not host mode, which only works on Linux). host.docker.internal isn't correct either since you're trying to reach another container, not your host
https://docs.docker.com/network/bridge/
I'd also recommend using Airflow Spark operators to actually read Parquet from HDFS, using Spark, not Pandas or WebHDFS. You can convert Spark dataframes to Pandas, if needed
I am trying to rerun the code I had written about 2 years ago but now it is not working. I am using python 3.7.1.
The code snippet causing the problem is following
import ssl
import mechanize
import operator
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
ssl._DEFAULT_CIPHERS = ('DES-CBC3-SHA')
myBrowser = mechanize.Browser()
myBrowser.set_handle_robots(False)
myBrowser.set_handle_refresh(False)
myBrowser.open("https://uwp.puchd.ac.in/common/viewmarks.aspx")
This is causing an exception during the ssl handshake and showing only [Error 0]. The complete traceback is following-
Traceback (most recent call last):
File "/home/him/.local/lib/python3.7/site-packages/mechanize/_urllib2_fork.py", line 1179, in do_open
headers)
File "/home/him/anaconda3/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/him/anaconda3/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/him/anaconda3/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/him/anaconda3/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/home/him/anaconda3/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/home/him/anaconda3/lib/python3.7/http/client.py", line 1392, in connect
server_hostname=server_hostname)
File "/home/him/anaconda3/lib/python3.7/ssl.py", line 412, in wrap_socket
session=session
File "/home/him/anaconda3/lib/python3.7/ssl.py", line 853, in _create
self.do_handshake()
File "/home/him/anaconda3/lib/python3.7/ssl.py", line 1117, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 0] Error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "UIET Rank system.py", line 51, in <module>
myBrowser.open("https://uwp.puchd.ac.in/common/viewmarks.aspx")
File "/home/him/.local/lib/python3.7/site-packages/mechanize/_mechanize.py", line 253, in open
return self._mech_open(url_or_request, data, timeout=timeout)
File "/home/him/.local/lib/python3.7/site-packages/mechanize/_mechanize.py", line 283, in _mech_open
response = UserAgentBase.open(self, request, data)
File "/home/him/.local/lib/python3.7/site-packages/mechanize/_opener.py", line 193, in open
response = urlopen(self, req, data)
File "/home/him/.local/lib/python3.7/site-packages/mechanize/_urllib2_fork.py", line 375, in _open
'_open', req)
File "/home/him/.local/lib/python3.7/site-packages/mechanize/_urllib2_fork.py", line 363, in _call_chain
result = func(*args)
File "/home/him/.local/lib/python3.7/site-packages/mechanize/_urllib2_fork.py", line 1225, in https_open
return self.do_open(conn_factory, req)
File "/home/him/.local/lib/python3.7/site-packages/mechanize/_urllib2_fork.py", line 1182, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 0] Error>
Can you please explain what is the problem actually and how to resolve it?
Note- Kindly Visit this link which i am opening and check, it uses outdated ciphers probably and it always shows warning before proceeding.
While it is not fully clear which version of OpenSSL you use together with Python my guess is that it is OpenSSL 1.1.1 since this gets shipped with Anaconda Python 3.7. This version does not support DES-CBC3-SHA anymore, i.e. this cipher is considered weak and not compiled in. That's why ssl._DEFAULT_CIPHERS = ('DES-CBC3-SHA') essentially sets the list of supported ciphers to none which makes the TLS handshake fail.
If you need this weak cipher than you need to go back to an older version of OpenSSL or recompile OpenSSL and Python with support for this weak cipher explicitly enabled.
I'm trying to get a Flask/Neo4j app set up on a remote Ubuntu server, and I've run into a problem that I haven't been able to figure out. My app uses py2neo, but when it tries to connect to the graph, the app crashes and the Neo4j process seems to stop. I've tried connecting in a python shell like this...
test = Graph('http://localhost:7474/db/data/',username='neo4j',password='myPassword')
which fails, and also renders neo4j inoperative until I restart it. However, these return 200 responses (and the web interface also works):
curl -u neo4j http://localhost:7474/db/data/
requests.get('http://localhost:7474/db/data/', auth=('neo4j','myPassword'))
I've tried to provide more information than this similar question, because it seems like the connection works from everywhere but py2neo.
Here's the full traceback:
Traceback (most recent call last):
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/database/__init__.py", line 318, in __new__
inst = cls.__instances[key]
KeyError: (<class 'py2neo.database.Graph'>, <ServerAddress settings={'http_port': 7474, 'host': 'localhost'}>, 'data')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 322, in submit
response = send()
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 317, in send
http.request(xstr(method), xstr(uri.absolute_path_reference), body, headers)
File "/usr/lib/python3.5/http/client.py", line 1106, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/usr/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 80, in connect
self.source_address)
File "/usr/lib/python3.5/socket.py", line 711, in create_connection
raise err
File "/usr/lib/python3.5/socket.py", line 702, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/database/__init__.py", line 327, in __new__
use_bolt = version_tuple(inst.__remote__.get().content["neo4j_version"]) >= (3,)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/database/http.py", line 154, in get
response = self.__base.get(headers=headers, redirect_limit=redirect_limit, **kwargs)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 966, in get
return self.__get_or_head("GET", if_modified_since, headers, redirect_limit, **kwargs)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 943, in __get_or_head
return rq.submit(redirect_limit=redirect_limit, **kwargs)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 433, in submit
http, rs = submit(self.method, uri, self.body, self.headers)
File "/home/deploy/toponimika/toponimikaenv/lib/python3.5/site-packages/py2neo/packages/httpstream/http.py", line 362, in submit
raise SocketError(code, description, host_port=uri.host_port)
py2neo.packages.httpstream.http.SocketError: Connection refused
Anything I might try to figure out what's going on would be appreciated.
Changed to http://username:password#localhost:7474/db/data/ and it works!
Example:
test = Graph('http://username:password#localhost:7474/db/data/')
I had the same issue, solved with a simple upgrade of pip version.
pip install --upgrade py2neo
I am using Python 3.4 on Windows 7, and I am trying to test if a proxy allows or denies connections to particular websites by using a python script.
I am using the code below:
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError, urllib
conf = "http://{}:{}#{}".format(login, password, proxy)
supp = urllib.request.ProxyHandler({"http": conf})
auth = urllib.request.HTTPBasicAuthHandler()
open = urllib.request.build_opener(supp, auth, urllib.request.HTTPHandler)
urllib.request.install_opener(open)
response = urlopen(Request("http://www.google.com"))
I get no errors when executing the code above, however as soon as I switch the URL to an HTTPS one (for example, https://www.google.com), I get the following error:
C:\Python34\python.exe test_url.py
Traceback (most recent call last):
File "C:\Python34\lib\urllib\request.py", line 1182, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "C:\Python34\lib\http\client.py", line 1088, in request
self._send_request(method, url, body, headers)
File "C:\Python34\lib\http\client.py", line 1126, in _send_request
self.endheaders(body)
File "C:\Python34\lib\http\client.py", line 1084, in endheaders
self._send_output(message_body)
File "C:\Python34\lib\http\client.py", line 922, in _send_output
self.send(msg)
File "C:\Python34\lib\http\client.py", line 857, in send
self.connect()
File "C:\Python34\lib\http\client.py", line 1223, in connect
super().connect()
File "C:\Python34\lib\http\client.py", line 834, in connect
self.timeout, self.source_address)
File "C:\Python34\lib\socket.py", line 494, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "C:\Python34\lib\socket.py", line 533, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11004] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 14, in <module>
response = urlopen(Request("https://www.google.com"))
File "C:\Python34\lib\urllib\request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 463, in open
response = self._open(req, data)
File "C:\Python34\lib\urllib\request.py", line 481, in _open
'_open', req)
File "C:\Python34\lib\urllib\request.py", line 441, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 1225, in https_open
context=self._context, check_hostname=self._check_hostname)
File "C:\Python34\lib\urllib\request.py", line 1184, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11004] getaddrinfo failed>
Any ideas why my code is only working with HTTP websites?
You need to specify the proxy handler for HTTPS separately, as it is a different protocol to HTTP. So the ProxyHandler line should be changed to:
supp = urllib.request.ProxyHandler({"http": conf, "https": conf})
I am new to python and I learned there are no different between single and double quoted string.
But I have found some different behavior.
from bs4 import BeautifulSoup
import urllib.request
url1 = "http://www.backpackers.com.tw/forum/forumdisplay.php?f=310"
url2 = 'http://www.backpackers.com.tw/forum/forumdisplay.php?f=310'
If I run:
response = urllib.request.urlopen(url1)
Result: Script finished without error
And If I run:
response = urllib.request.urlopen(url2)
Result: error
C:\Users\user1\Desktop\scrape>python backpacker_tw.py
Traceback (most recent call last):
File "C:\Python34\lib\urllib\request.py", line 1189, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "C:\Python34\lib\http\client.py", line 1090, in request
self._send_request(method, url, body, headers)
File "C:\Python34\lib\http\client.py", line 1128, in _send_request
self.endheaders(body)
File "C:\Python34\lib\http\client.py", line 1086, in endheaders
self._send_output(message_body)
File "C:\Python34\lib\http\client.py", line 924, in _send_output
self.send(msg)
File "C:\Python34\lib\http\client.py", line 859, in send
self.connect()
File "C:\Python34\lib\http\client.py", line 836, in connect
self.timeout, self.source_address)
File "C:\Python34\lib\socket.py", line 509, in create_connection
raise err
File "C:\Python34\lib\socket.py", line 500, in create_connection
sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the
target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "backpacker_tw.py", line 7, in <module>
response = urllib.request.urlopen(url2)
File "C:\Python34\lib\urllib\request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 455, in open
response = self._open(req, data)
File "C:\Python34\lib\urllib\request.py", line 473, in _open
'_open', req)
File "C:\Python34\lib\urllib\request.py", line 433, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 1215, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python34\lib\urllib\request.py", line 1192, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [WinError 10061] No connection could be ma
de because the target machine actively refused it>
Is it a bug or anything i missed?
C:\Users\user1\Desktop\scrape>python -V
Python 3.4.1
To the documentation! PEP 8, pretty much the be all end all on python code formatting, states "In Python, single-quoted strings and double-quoted strings are the same." This is written by the creator of Python, and I'm going to take his word for it.
Looking at your stack trace I see the error No connection could be ma
de because the target machine actively refused it, so maybe that means something wonky was going on with the server at that point?