Develop Reference

Python/P2P - Unable to connect to rendezvous server

I am trying to create a P2P node using python (pyp2p) but I am getting this error:
Eamons-MacBook-Pro:blockchain eamonwhite$ python3 serveralice.py
HTTP Error 404: Not Found
HTTP Error 404: Not Found
HTTP Error 404: Not Found
HTTP Error 404: Not Found
Traceback (most recent call last):
File "/Users/eamonwhite/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pyp2p/net.py", line 732, in start
rendezvous_con = self.rendezvous.server_connect()
File "/Users/eamonwhite/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pyp2p/rendezvous_client.py", line 92, in server_connect
con.connect(server["addr"], server["port"])
File "/Users/eamonwhite/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pyp2p/sock.py", line 189, in connect
self.s.bind((src_ip, 0))
TypeError: str, bytes or bytearray expected, not NoneType
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "serveralice.py", line 10, in <module>
alice.start()
File "/Users/eamonwhite/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pyp2p/net.py", line 735, in start
raise Exception("Unable to connect to rendezvous server.")
Exception: Unable to connect to rendezvous server.
My relevant code looks like this:
from uuid import uuid4
from blockchain import Blockchain
from flask import Flask, jsonify, request
from pyp2p.net import *
import time
#Setup Alice's p2p node.
alice = Net(passive_bind="192.168.1.131", passive_port=44444, interface="en0", node_type="passive", debug=1)
alice.start()
alice.bootstrap()
alice.advertise()
while 1:
for con in alice:
for reply in con:
print(reply)
time.sleep(1)
...
It is getting stuck on the Net function right at the beginning - something to do with the rendezvous package. The IP is my IP on the my network, and I port forwarded 44444 although I'm not sure if I need to do that or not. Thanks.

I am new to this, apparently with the way the server code was configured, it needed a rendezvous server to work (a node that handles all the other nodes). It is in net.py of the pyp2p package:
# Bootstrapping + TCP hole punching server.
rendezvous_servers = [
{
"addr": "162.243.213.95",
"port": 8000
}
]
The address was the problem, obviously it is just a placeholder IP. So then I realized I needed my own rendezvous server, and I used this code - https://raw.githubusercontent.com/StorjOld/pyp2p/master/pyp2p/rendezvous_server.py.
However I had to debug this file a little, it ended up needing to have import sys, import time and import re statements at the top before it would work. Now I am going to host it on my raspberry pi so that it is always up to handle nodes :)

python and kafka KeyError: -1

Python kafka does not work. Any reason why? I mean I cant event connect to kafka and i get an error? Is there a kafka client that works with 0.8? I mean this is a brand new server. I just booted it up.
I am using https://github.com/mumrah/kafka-python
from kafka.client import KafkaClient
kafka = KafkaClient(kafka_domain, 9092)
Traceback (most recent call last):
File "/home/ubuntu/workspace/rtbhui-devops/simulations/pixel_druid_simulations.py", line 36, in <module>
kafka = KafkaClient(kafka_domain, 9092)
File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 38, in __init__
self.load_metadata_for_topics() # bootstrap with all metadata
File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 247, in load_metadata_for_topics
self.topics_to_brokers[topic_part] = brokers[meta.leader]
KeyError: -1
In kafka logs I see the below.
[2014-02-26 08:36:21,471] INFO Closing socket connection to /222.127.xxx.xxx. (kafka.network.Processor)
[2014-02-26 08:40:30,801] ERROR [KafkaApi-1393401480] Error while fetching metadata for partition [topic-pixel,0] (kafka.server.KafkaApis)
kafka.common.LeaderNotAvailableException: Leader not available for partition [topic-pixel,0]
at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:474)
at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:462)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
at scala.collection.immutable.List.foreach(List.scala:45)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.List.map(List.scala:45)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:462)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:458)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.Set$Set1.map(Set.scala:68)
at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:458)
at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:744)

This is fixed in kafka-python version 0.9.0. The error is internal to Kafka Python, not the kafka server, and simply involves handling partitions that are currently without a leader (which happens for any new topic when you auto-create topics, but other than that is fairly rare under normal operations)
See
https://github.com/mumrah/kafka-python/pull/109

ERROR in using python to read data from hbase through thrift

I am trying to read data from hbase use python. I installed thrift, and generated the gen-py files then moved it to the python lib:~/.local/lib/python2.7/site-packages/
The hbase thrift server is in 192.168.15.116:39090. Is was started.
my code is in the server 192.168.15.146. In this machine, I can use hbase shell command to read the hbase data.
Here is my Python code:
#! /usr/bin/env python
from thrift import Thrift
from thrift.transport import TSocket, TTransport
from thrift.protocol import TBinaryProtocol
from hbase import Hbase
transport = TSocket.TSocket('192.168.15.116', 39090)
transport.setTimeout(5000)
#transport = TTransport.TBufferedTransport(transport)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = Hbase.Client(protocol)
transport.open()
print(client.getTableNames())
It was all ok but the last line, here is the error:
Traceback (most recent call last):
File "test01.py", line 24, in <module>
client.getTableNames()
File "~/.local/lib/python2.7/site-packages/hbase/Hbase.py", line 788, in getTableNames
return self.recv_getTableNames()
File "~/.local/lib/python2.7/site-packages/hbase/Hbase.py", line 803, in recv_getTableNames
raise x
thrift.Thrift.TApplicationException: Internal error processing getTableNames
I googled it but could'nt find the way to solve this ERRO, could anyone give me some help?
Thanks in advance!

I'm not sure but after having a quick look in the code it seems that the handler/processor code on the Server throws an unexpected exception. Unexpected meaning: neither a thrift one, nor one of the declared exceptions in the IDL file.
In that case the recommendation would be to look at HBase, whether this is a known issue, etc.

Please check your thrift host and Port, you can use happybase, example:
import happybase
connection = happybase.Connection(host='hadoop_env.com',port=9090,timeout=1000000)
connection.open()
print connection.tables()
There is another series of example:
https://github.com/Shadow-Hunter-X/python_practice_stepbystep/blob/master/python-on-bigdata/chapter6/chapter6_happybase.py

Connection refused to Twitter API on PythonAnywhere

I am trying to connect to the twitter streaming API on Python anywhere, but always get a connection refused error.
I use Tweepy in my application, and to test the connection I am using the streaming example that can be found in the repo.
HEre is a sum-up of the code :
from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream
# Go to http://dev.twitter.com and create an app.
# The consumer key and secret will be generated for you after
consumer_key=""
consumer_secret=""
# After the step above, you will be redirected to your app's page.
# Create an access token under the the "Your access token" section
access_token=""
access_token_secret=""
class StdOutListener(StreamListener):
""" A listener handles tweets are the received from the stream.
This is a basic listener that just prints received tweets to stdout.
"""
def on_data(self, data):
print data
return True
def on_error(self, status):
print status
if __name__ == '__main__':
l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
stream.filter(track=['basketball'])
When I run this line in a bash console in python anywhere (after having filled the tokens of course)
12:02 ~/tweepy/examples (master)$ python streaming.py
I get the following error :
Traceback (most recent call last):
File "streaming.py", line 33, in <module>
stream.filter(track=['basketball'])
File "/usr/local/lib/python2.7/site-packages/tweepy/streaming.py", line 228, in filter
self._start(async)
File "/usr/local/lib/python2.7/site-packages/tweepy/streaming.py", line 172, in _start
self._run()
File "/usr/local/lib/python2.7/site-packages/tweepy/streaming.py", line 106, in _run
conn.connect()
File "/usr/local/lib/python2.7/httplib.py", line 1157, in connect
self.timeout, self.source_address)
File "/usr/local/lib/python2.7/socket.py", line 571, in create_connection
raise err
socket.error: [Errno 111] Connection refused
The domain .twitter.com is in the pythonanywhere whithelist though, so i don't understand why the connection would be refused :s.
The very same code works like a charm on my Ubuntu.
Any idea would be more than welcome, thanks !!

If you're using a Free account, tweepy won't work. It does not use the proxy settings from the environment.
There is a fork of tweepy that you might be able to use (http://github.com/ducu/tweepy) until the main line uses the proxy settings correctly.

As Glenn said, there currently isn't proxy support in tweepy.
For a reason I cannot explain (and isn't documented), a pull request was closed without any merge about a month ago.
https://github.com/tweepy/tweepy/pull/152
There apparently is a fork available on github (see Glenn's answer), but I didn't test it.
Knowing that I would need to use my own domain name in the end, I finally got a paid account on pythonanywhere and got rid of the proxy stuff all together.

Using GAE remote api for debugging from localhost - Connecting too late?

Trying to use Google App Engine's remote_api so that we can do line-by-line debugging through the IDE.
The remote api works great at first. The application is able to successfully retrieve information from the database. The error occurs when wepapp responds to the client browser.
The Code:
It is very similar to the example given in app engine's documentation:
from model import My_Entity
from google.appengine.ext.remote_api import remote_api_stub
# Test database calls
def get(w_self):
remote_api_stub.ConfigureRemoteApi(None, '/_ah/remote_api', auth_func, 'myapp.appspot.com')
t_entity = My_Entity.get_by_key_name('the_key')
w_self.response.set_status(200)
# The error occurs AFTER this code executes, when webapp actually responds to the browser
Error Traceback:
The error seems to be related to the blobstore.
Is the remote api initialized too late into the code?
...After webapp has done something with the blobstore through the localhost server? Then the remote api might be re-directing requests to the blobstore in the server instead of the localhost debug server where webapp is expecting it to be?
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 2795, in _HandleRequest
login_url)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 3622, in CreateImplicitMatcher
get_blob_storage)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_blobstore.py", line 420, in CreateUploadDispatcher
return UploadDispatcher()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_blobstore.py", line 307, in __init__
get_blob_storage())
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver_blobstore.py", line 79, in GetBlobStorage
return apiproxy_stub_map.apiproxy.GetStub('blobstore').storage
AttributeError: 'RemoteStub' object has no attribute 'storage'
Should the remote api be initialized somewhere else in the code?
Or does this problem have to do with something else?
Thanks so much!

To get this working you can use the help of the testbed to start the stubs that are missing:
ADDRESS=....
remote_api_stub.ConfigureRemoteApi(None, '/_ah/remote_api', auth_func, ADDRESS)
# First, create an instance of the Testbed class.
myTestBed = testbed.Testbed()
# Then activate the testbed, which prepares the service stubs for use.
myTestBed.activate()
# Next, declare which service stubs you want to use.
myTestBed.init_blobstore_stub()
myTestBed.init_logservice_stub()