I am trying to write on HDFS from Python.
Right now, I am using https://hdfscli.readthedocs.io/en/latest/quickstart.html
but for large file I get back:
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 400, in write
consumer(data)
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 394, in consumer
auth=False,
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 179, in _request
**kwargs
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))
My code for writing is pretty simple:
client = InsecureClient('http://xxxxxxx.co:50070', user='hdfs')
client.write("/tmp/a",stringToWrite)
Anyone can suggest a decent package to write on HDFS?
Cheers
For the stacktrace, it seems to be security related. Are you sure you need to use the InsecureClient and not the Kerberos one?. Also, remember that library is just a binding for HttpFs, so doing a manual test with Postman or CURL would let you debug any issue cluster-side.
Related
I use instabot to upload images on IG. The problem, in general, is that Instagram blocks my bot and I always have to use a code sent to mail email to reload my account. I know that IG does not like bots instabot is from 3 bots just one who is working. but it has still this issue. I tried to make a bot using Instagram-private-API but a way to get a working API on Instagram is so long and from the past, I know the use of APIs of big companies is really a complete task and I want to make it easy in some way how to make it prevent instant or some better alternative that Instagram official APIs.
my code:
from instabot import Bot
def upload_to_instagram(caption):
# Create an InstagramBot object
bot = Bot()
bot.login(username= "nature_for_best_day",
password="""29$5!5U9Ba%C$#M6""")
# Upload the image
media_id = bot.upload_photo("image_with_quote.jpg",caption)
# Set the caption for the image
print("Photo uploaded")
my error:
File "c:\Users\Lukas\Dokumenty\python_scripts\Instagram Quotes\main.py", line 92, in upload_to_instagram
media_id = bot.upload_photo("image_with_quote.jpg",caption)
File "C:\Users\Lukas\Dokumenty\python_scripts\Instagram Quotes\env\bot\lib\site-packages\instabot\bot\bot.py", line 806, in upload_photo
return upload_photo(self, photo, caption, upload_id, from_video, options)
File "C:\Users\Lukas\Dokumenty\python_scripts\Instagram Quotes\env\bot\lib\site-packages\instabot\bot\bot_photo.py", line 26, in upload_photo
result = self.api.upload_photo(
File "C:\Users\Lukas\Dokumenty\python_scripts\Instagram Quotes\env\bot\lib\site-packages\instabot\api\api.py", line 825, in upload_photo
return upload_photo(
File "C:\Users\Lukas\Dokumenty\python_scripts\Instagram Quotes\env\bot\lib\site-packages\instabot\api\api_photo.py", line 168, in upload_photo
response = self.session.post(
File "C:\Users\Lukas\Dokumenty\python_scripts\Instagram Quotes\env\bot\lib\site-packages\requests\sessions.py", line 635, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "C:\Users\Lukas\Dokumenty\python_scripts\Instagram Quotes\env\bot\lib\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\Lukas\Dokumenty\python_scripts\Instagram Quotes\env\bot\lib\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "C:\Users\Lukas\Dokumenty\python_scripts\Instagram Quotes\env\bot\lib\site-packages\requests\adapters.py", line 563, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='i.instagram.com', port=443): Max retries exceeded with url: /rupload_igphoto/1675010396498_0_9661884500 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2396)')))
I executed this code to connect most common api-
import requests
response = requests.get("http://api.open-notify.org/iss-now.json")
print(response.status_code)
But it is showing this error -
runfile('C:/Users/sanchit.joshi/use case of unassigned tickets/Api try
out.py', wdir='C:/Users/sanchit.joshi/use case of unassigned tickets')
Traceback (most recent call last):
File "<ipython-input-17-39bcdc5917ae>", line 1, in <module>
runfile('C:/Users/sanchit.joshi/use case of unassigned tickets/Api try
out.py', wdir='C:/Users/sanchit.joshi/use case of unassigned tickets')
File "C:\ProgramData\Anaconda3\lib\site-
packages\spyder_kernels\customize\spydercustomize.py", line 668, in
runfile
execfile(filename, namespace)
File "C:\ProgramData\Anaconda3\lib\site-
packages\spyder_kernels\customize\spydercustomize.py", line 108, in
execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/sanchit.joshi/use case of unassigned tickets/Api try
out.py", line 8, in <module>
response = requests.get("http://api.open-notify.org/iss-now.json")
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\api.py", line
72, in get
return request('get', url, params=params, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\api.py", line
58, in request
return session.request(method=method, url=url, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py",
line 512, in request
resp = self.send(prep, **send_kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py",
line 622, in send
r = adapter.send(request, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py",
line 513, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='api.open-notify.org',
port=80): Max retries exceeded with url: /iss-now.json (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x000001E8E5BCBE80>: Failed to establish a new connection: [Errno 11002]
getaddrinfo failed'))
I tried changing max retry value, but its not working. It is more frustrating bcz I this is the simplest code to connect to an api. Any help is appreciated.
There is nothing wrong with your code, that code should work.
You are having a proxy issue. If you are on windows you can add the url to your proxy exceptions by going to the settings menu in internet explorer, then internet options, then connections, lan settings advanced and add the url to your exceptions. It is typical in a corporate environment or in your school for the admins to put you behind a proxy.
Alternatively you can use this Q/A to set the proxy in your request
Using the JSON and urllib.request libraries
import json
import urllib.request
file = urllib.request.urlopen("http://api.open-notify.org/iss-now.json")
data = json.loads(file.read())
print(data)
Resulting in
{'message': 'success', 'timestamp': 1541059187, 'iss_position': {'longitude': '13.6813', 'latitude': '47.8641'}}
I'm trying to send an http get request via the httplib, but I'm facing issues.
conn = httplib.HTTPConnection("10.30.111.13/View")
conn.request("GET", "/Default.aspx")
res = conn.getresponse()
if res.status == 200:
print(res.status)
else:
print("Something went terribly wrong")
I get the following error:
TypeError (cannot concatenate 'str' and 'int' objects).
If put the next line of codes, it works no problem:
conn = httplib.HTTPConnection("www.google.com")
conn.request("GET", "/")
EDIT, here is a more detailed log I managed to pull out of my third party software (it restricts me in turn of python usability):
File "<string>", line 3248, in initialization
File "C:\python22\lib\httplib.py", line 701, in request
self._send_request(method, url, body, headers)
File "C:\python22\lib\httplib.py", line 723, in _send_request
self.endheaders()
File "C:\python22\lib\httplib.py", line 695, in endheaders
self._send_output()
File "C:\python22\lib\httplib.py", line 581, in _send_output
self.send(msg)
File "C:\python22\lib\httplib.py", line 548, in send
self.connect()
File "C:\python22\lib\httplib.py", line 516, in connect
socket.SOCK_STREAM):
gaierror: (7, 'getaddrinfo failed')
I'm not someplace where I can test this now, but here's what I think:
You're passing only an IP address to a host field that's expecting a DNS address, not an IP address. That's why your second error listing says 'getaddrinfo' failed.
That said, I'm not sure how to use an IP address with httplib. Maybe try "http://10.30.111.13" instead. A good way to test it would be to replace your IP address above with Google's and see if you still get the error.
Maybe this will help -- sorry I can't say more!
I have changed the IP address for a DNS address. I also removed any path/URI that were in the HTTPConnection() parameter. Now it works. Sorry for such an obvious question guys.
Hi I am new to Python...
I am trying to use YQL using Python.
I installed httplib2-0.7.0, oauth2 and then installed yql package
For this sample code :
import yql
y = yql.Public()
query = 'select * from flickr.photos.search where text="panda" limit 3';
result = y.execute(query)
print result
I got the following error message.
Please help!!
Traceback (most recent call last):
File "test.py", line 4, in
result = y.execute(query)
File "C:\Python27\lib\site-packages\yql-0.7-py2.7.egg\yql__init__.py", line 306, in execute
resp, content = self.http.request(url, http_method)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 1436, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey
)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 1188, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 1123, in _conn_request
conn.connect()
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 890, in connect
self.disable_ssl_certificate_validation, self.ca_certs)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 76, in _ssl_wrap_socket
cert_reqs=cert_reqs, ca_certs=ca_certs)
File "C:\Python27\lib\ssl.py", line 344, in wrap_socket
ciphers=ciphers)
File "C:\Python27\lib\ssl.py", line 119, in init
ciphers)
ssl.SSLError: [Errno 185090050] _ssl.c:336: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
When I tried to use the *twitter python wrapper, I ended up getting the same SSL error.*
Please tell me what to do
I can see two likely issues:
SSL Certificate
I'm not familiar with Python or that library you're using, but the error sounds like it can't verify the SSL certificate. (Possibly because there is no suitable local SSL certificate bundle for authentication.) You may be able to configure it to skip the SSL certificate verification.
YQL Flickr Query
The YQL query is not correct and gives an error in the YQL console:
select * from flickr.photos.search where text="panda" limit 3
Actually, given the age of this question, it may have worked in June 2011. Now the Flickr tables require an API key as well, so the working query would look like:
select * from flickr.photos.search where text="panda" and api_key="insert-your-key-here" limit 3
I'm trying to use httplib to send credit card information to authorize.net. When i try to post the request, I get the following traceback:
File "./lib/cgi_app.py", line 139, in run res = method()
File "/var/www/html/index.py", line 113, in ProcessRegistration conn.request("POST", "/gateway/transact.dll", mystring, headers)
File "/usr/local/lib/python2.7/httplib.py", line 946, in request self._send_request(method, url, body, headers)
File "/usr/local/lib/python2.7/httplib.py", line 987, in _send_request self.endheaders(body)
File "/usr/local/lib/python2.7/httplib.py", line 940, in endheaders self._send_output(message_body)
File "/usr/local/lib/python2.7/httplib.py", line 803, in _send_output self.send(msg)
File "/usr/local/lib/python2.7/httplib.py", line 755, in send self.connect()
File "/usr/local/lib/python2.7/httplib.py", line 1152, in connect self.timeout, self.source_address)
File "/usr/local/lib/python2.7/socket.py", line 567, in create_connection raise error, msg
gaierror: [Errno -2] Name or service not known
I build my request like so:
mystring = urllib.urlencode(cardHash)
headers = {"Content-Type": "text/xml", "Content-Length": str(len(mystring))}
conn = httplib.HTTPSConnection("secure.authorize.net:443", source_address=("myurl.com", 443))
conn.request("POST", "/gateway/transact.dll", mystring, headers)
to add another layer to this, it was working on our development server which has httplib 2.6 and without the source_address parameter in httplib.HTTPSConnection.
Any help is greatly appreciated.
===========================================================
EDIT:
I can run it from command line. Apparently this is some sort of permissions issue. Any ideas what permissions I would need to grant to which users to make this happen? Possibly Apache can't open the port?
As an (obvious) heads up, this same error can also be triggered by including the protocol in the host parameter. For example this code:
conn = httplib.HTTPConnection("http://secure.authorize.net", 80, ....)
will also cause the "gaierror: [Errno -2] Name or service not known" error, even if all your networking setup is correct.
gaierror: [Errno -2] Name or service not known
This error often indicates a failure of your DNS resolver. Does ping secure.authorize.net return successful replies from the same server that receives the gaierror? Does the hostname have a typo in it?
The problem ultimately came down to the fact that selinux was stopping apache from getting that port. Disabling selinux fixed the problems. I had an issue later where i didn't have /var/www/.python-eggs/, so MySQLdb was hosing on import. But after a mkdir, it was fixed.
pass the port separately from the host:
conn = httplib.HTTPSConnection("secure.authorize.net", 443, ....)