Develop Reference

Can someone suggest alternative to HdfsSensor for airflow python3?

I am trying to listen to changes in HDFS to trigger my ETL pipeline in Airflow using HdfsSensor in python3. I am getting the following error as snakebite is not supported for python3
This HDFSHook implementation requires snakebite, but '
ImportError: This HDFSHook implementation requires snakebite, but snakebite is not compatible with Python 3

Thanks to the suggestion by #AyushGoyal, I solved the same problem using WebHDFSSensor. This sensor looks like HdfsSensor and you can just replace the function names. just remember to make sure:
you pass the connection id via webhdfs_conn_id parameter (in HdfsSensor the parameter name was hdfs_conn_id)
the port with which you should try to connect to name node is 50700 (not 8020)
The rest is the same!
example:
from airflow.sensors.web_hdfs_sensor import WebHdfsSensor
file_sensor = WebHdfsSensor(
task_id='check_if_data_is_ready',
filepath="some_file_path",
webhdfs_conn_id='hdfs_conn_id',
poke_interval=10,
timeout=5,
dag=dag,
env={
'JAVA_HOME': '/usr/java/latest'
}
)

pyspeedtest cannot find test server

I'm trying to use pyspeedtest to get the upload/download speed of my connecting but I keep getting the following error which I couldn't resolve:
import pyspeedtest
st = pyspeedtest.SpeedTest()
st.download()
Exception: Cannot find a test server
Any suggestions/insights would be welcome!

It actually does work if you change the url in the pyspeedtest.py file from www.speedtest.net to c.speedtest.net on line 186 in v1.2.7 of the script.
Edit: added an example of how to get it to work
You can edit the pyspeedtest.py script (located at /usr/local/lib/python2.7/dist-packages/pyspeedtest.py on my raspberry pi 3) by using vi, e.g.:
sudo vi /usr/local/lib/python2.7/dist-packages/pyspeedtest.py
Go to line 186 and change the following line:
connection = self.connect('www.speedtest.net')
to:
connection = self.connect('c.speedtest.net')
Then run pyspeedtest using the wrapper in /usr/local/bin:
/usr/local/bin/pyspeedtest
Using server: speedtest.wilkes.net
Ping: 41 ms
Download speed: 46.06 Mbps
Upload speed: 11.58 Mbps
Or use the python interpreter:
>>> import pyspeedtest
>>> st = pyspeedtest.SpeedTest()
>>> st.ping()
41.70024394989014
>>> st.download()
44821310.72337018
>>> st.upload()
14433296.732646577

The project hasn't been updated since mid-2016. And the last update was updated user-agent to prevent SpeedTest block... And if you skim the code, there are [comments like this]:(https://github.com/fopina/pyspeedtest/blob/master/pyspeedtest.py#L188)
# really contribute to speedtest.net OS statistics
# maybe they won't block us again...
And there have been bugs posted to GitHub about the project not working, with no response.
So, my guess is: This project probably violates SpeedTest.net's terms of service, so they blocked it. The author tried to get around the block, they blocked it again, and the author gave up. In the intervening two years, any other servers it used as backups either blocked it, or shut down (e.g., speedtest.serv.pt, mentioned in the docs, no longer exists).
There is a pull request from another user that might fix it, although it appears to be failing the CI test. If you want to try it yourself, you can.
But otherwise, you can't use this library, and there's no way anyone can help you use it; it just doesn't work. You'll have to find another way to do the same thing.

Pyro requests goes to the wrong nameserver (when there are multiple ones)

I have the following case:
Two servers running each their own name server (NS)
Each of these servers has the same object registering with a different URI. The URI includes the local server's hostname to make it unique
A third server, the client, tries to target the right server to query information available only that server
Please note that all of these 3 servers can communicate with each other.
My questions and issues:
The requests from the third server always goes to the first server, no matter what; except when I shutdown the first NS. Is there something definitely wrong with that I'm doing? I guess I do, but I can't figure it out...
Is running separate nameservers the root cause? What would be the alternative if this not allowed? I run multiple name servers for redundancy as some other upcoming operations can run on any of the two first servers. When I list the content of each name server (locally on each server), I get the right registration (which includes the hostname).
Is the use of Pyro4.config.NS_HOST parameter wrong, see below the usage in the code? What would be the alternative?
My configuration:
Pyro 4-4.63-1.1
Python 2.7.13
Linux OpenSuse (kernel version 4.4.92)
The test code is listed below. I got rid of the details like try blocks and imports, etc...
My server skeleton code (which runs on the first two servers):
daemon = Pyro4.Daemon(local_ip_address)
ns = Pyro4.locateNS()
uri = daemon.register(TestObject())
ns.register("test.object#%s" % socket.gethostname(), uri)
daemon.requestLoop()
The local_ip_address is the one supplied below by the user to contact the correct name server (on the correct server).
The name server is started on each of the first tow servers as follows:
python -m Pyro4.naming -n local_ip_address
The local_ip_address is the same as above.
My client skeleton code (which runs on the third server):
target_server_hostname = user_provided_hostname
target_server_ip = user_provided_ip
Pyro4.config.NS_HOST = target_server_ip
uri = "test.object#%s" % target_server_hostname
proxy = Pyro4.Proxy(uri)
proxy._pyroTimeout = my_timeout
proxy._pyroMaxRetries = my_retries
rc, reason = proxy.testAction(target_server_hostname)
if rc != 0:
print reason
else:
print "Hostname matches"
If more information is required, please let me know.
Djurdjura.

I think figured it out. Hope this will be useful to anyone else looking for a similar use case.
You just need to specify where to look for the name server itself. The client code becomes something like the following:
target_server_hostname = user_provided_hostname
target_server_ip = user_provided_ip
# The following statement finds the correct name server
ns = Pyro4.locateNS(host=target_server_ip)
name = "test.object#%s" % target_server_hostname
uri = ns.lookup(name)
proxy = Pyro4.Proxy(uri)
proxy._pyroTimeout = my_timeout
proxy._pyroMaxRetries = my_retries
rc, reason = proxy.testAction(target_server_hostname)
if rc != 0:
print reason
else:
print "Hostname matches"
In my case, I guess the alternative would be using a single common name server running on ... the third server (the client server). This server is always on and ready before the other ones. I didn't try this approach one yet.
Regards.
D.
PS. Thanks Irmen for your answer.

Proper version of boto for Eucalyptus cloud

I'm writing some code to interact with an HP Helion Eucalyptus 4.2 cloud server.
At the moment I'm using boto 2.38.0, but I discovered that also exists
the boto3 version.
Which version should I use in order to keep the code up with the times?
I mean, It seems that boto3's proposal is a ground-up rewrite more focused
on the "official" Amazon Web Services (AWS).

You can easilly use boto3. Here is the article clearly explaining that.
I tried it myself with Eucalyptus 4.2.1. So I defined the config and credentials file.
$ cat .aws/config
[profile admin]
output = json
region = region1
$ cat .aws/credentials
[admin]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Then I used the interactive Ipython shell, so I listed my running instances.
In [1]: from boto3.session import Session
In [2]: session = Session(region_name='region1', profile_name="admin")
In [3]: client = session.client('ec2', endpoint_url='http://compute.region1.cloud.mydomain.net:8000/')
In [4]: for reservation in client.describe_instances()['Reservations']:
...: for instance in reservation['Instances']:
...: print instance['InstanceId']
...:
i-c706792f
i-af613099
i-fc0c55ee
i-4f493764
i-943d0ce3
I don't use verify argument in session.client(), because my testing EC2 endpoint doesn't use HTTPS but plain HTTP.

2.38 is the right version. boto3 is something totally different and I don't have experience with it.

Check Domain Availability using Boto Route53

I love using Boto API for Amazon Web Services but now I'm not capable of finding where is the error.
I'm using AWS for check domain availability and I have created a script in Python that includes the class at this link:
https://www.codatlas.com/github.com/boto/boto/develop/boto/route53/domains/layer1.py?line=67
I call the method check_domain_availability() on passing domain name:
Route53DomainsConnection.check_domain_availability('example.com',None)
but the method returns this error:
AttributeError: 'str' object has no attribute 'make_request'
I can try to pass parameters in many modes but no result.
Where am I wrong? Thanks in advance.
P.S: I use Debian wheezy and Python3.2
More on status of subdomains
I have found a method to get the status of a record just create with route53.
this is the code:
changes = ResourceRecordSets(conn, "ZONEID")
change = changes.add_change("STRING FOR ADD NEW SUBDOMAIN")
change.add_value(MY_IP)
status = changes.commit()
If print the status variable is contained the response of commit and the status:
{u'ChangeResourceRecordSetsResponse':{u'ChangeInfo': {u'Status: u'PENDING etc.....
Now i would like to be able to swhitch to another operation only if the status of subdomamin is "SYNC" but i doesn't able to access dinamically to string for check status.
I can use a while ? Can i use sleep command ? Can anyone help me over to resolve my problem ? Thanks

You don't show your code which makes it harder to debug but this line:
Route53DomainsConnection.check_domain_availability('example.com',None)
looks suspicious. It looks like you are trying to access the check_domain_availability method from the class rather than an instance of the class. I just did the following and it worked for me:
In [1]: import boto.route53.domains
In [2]: c = boto.route53.domains.connect_to_region('us-east-1')
In [3]: c.check_domain_availability('foobar.com')
Out[3]: {u'Availability': u'UNAVAILABLE'}