Develop Reference

Can't initialize ANT+ Node with Python OpenANT library

I've totally new in Python and also in the ANT+ technology. I wonder if that's not some basic problem, but I've been strugling with it for couple of days already browsing through forums with no luck..
So I'm trying to use the Python OpenANT library (https://github.com/Tigge/openant) to access my ANT doungle which is plugged into the USB port (WINDOWS 10 PRO). My goal is to access my Garmin through it and get some data from it. However, I'm stuck at the very beginning trying to inizialize the ANT Node. My code is this:
from ant.easy.node import Node
node=Node()
To this I get the exception:
File "C:/Users/Edgars/Desktop/untitled-5.py", line 2, in <module>
pass
File "C:\Users\Edgars\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\ant\easy\node.py", line 56, in __init__
self.ant = Ant()
File "C:\Users\Edgars\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\ant\base\ant.py", line 68, in __init__
self._driver.open()
File "C:\Users\Edgars\AppData\Local\Programs\Python\Python38-32\Lib\site-packages\ant\base\driver.py", line 193, in open
cfg = dev.get_active_configuration()
File "C:\Users\Edgars\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyusb-1.1.0-py3.8.egg\usb\core.py", line 909, in get_active_configuration
return self._ctx.get_active_configuration(self)
File "C:\Users\Edgars\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyusb-1.1.0-py3.8.egg\usb\core.py", line 113, in wrapper
return f(self, *args, **kwargs)
File "C:\Users\Edgars\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyusb-1.1.0-py3.8.egg\usb\core.py", line 250, in get_active_configuration
bConfigurationValue=self.backend.get_configuration(self.handle)
File "C:\Users\Edgars\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyusb-1.1.0-py3.8.egg\usb\backend\libusb0.py", line 519, in get_configuration
ret = self.ctrl_transfer(
File "C:\Users\Edgars\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyusb-1.1.0-py3.8.egg\usb\backend\libusb0.py", line 601, in ctrl_transfer
return _check(_lib.usb_control_msg(
File "C:\Users\Edgars\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyusb-1.1.0-py3.8.egg\usb\backend\libusb0.py", line 447, in _check
raise USBError(errmsg, ret)
usb.core.USBError: [Errno None] b'libusb0-dll:err [control_msg] sending control message failed, win error: A device which does not exist was specified.\r\n\n'
I have closed the Garmin Agent, so no other programs are using my ANT dongle at the same time. When I run my code, the specific sound occurs every time - the one that we hear when we detach a USB device by selecting "Eject" from the drop-down menu (the sound happens simultaneously with the exception message), so I guess the USB gets accessed at some moment.
Before the exception I get such a printout:
Driver available: [<class 'ant.base.driver.SerialDriver'>, <class 'ant.base.driver.USB2Driver'>, <class 'ant.base.driver.USB3Driver'>]
- Using: <class 'ant.base.driver.USB3Driver'>
Could not check if kernel driver was active, not implemented in usb backend
I have seen other users' threads where the printout says Using ... USB1Driver or Using ... USB2Driver, and they don't get this message. I've installed various python libraries trying to get even this far, and now I've worried that maybe they get in each other's way.. Can anybody help me with this? It's really frustrating that a program of only two code lines can get so complicated.. :D
!!!EDIT!!!
OK, I found the problem - in the "driver.py" file there's a line dev.reset() which disconnects my USB dongle before trying to access it. I have no idea why such a line should exist there. I tried to comment this line out, and now I'm not getting the abovementioned error anymore. However, what happens now is there are continuos timeouts..
So my code has evolved to this (although actually the same timeouts happen also with my initial 2-lines-long program):
from ant.easy.node import Node
from ant.easy.channel import Channel
from ant.base.message import Message
import threading
NETWORK_KEY=[0xb9,0xa5,0x21,0xfb,0xbd,0x72,0xc3,0x45]
def on_data(data):
print("Data received")
print(data)
def back_thread(node):
node.set_network_key(0x00,NETWORK_KEY)
channel=node.new_channel(Channel.Type.BIDIRECTIONAL_RECEIVE)
channel.on_broadcast_data=on_data
channel.on_burst_data=on_data
channel.set_period(16070)
channel.set_search_timeout(20)
channel.set_rf_freq(57)
channel.set_id(0,120,0)
try:
channel.open()
node.start()
finally:
node.stop()
print("ANT Node Shutdown Complete")
node=Node()
x=threading.Thread(target=back_thread,args=(node,))
x.start()
Now I get this error line printed out for ever:
<class 'usb.core.USBError'>, (None, b'libusb0-dll:err [_usb_reap_async] timeout error\n')
When my Garmin Agent is active, I get the error "ANT resource already in use" instead of the timeout, so I'm certain that my code is accessing the ANT dongle.. However, now (having closed the Garmin Agent) I have no idea about how to get rid of the timeout and how to establish a simple handshake with my Garmin device..

OK, now I've figured out that my Garmin Forerunner 310XT can't act as a data source and thus cannot be accessed using the ANT+ protokol. Instead, I should use the ANT-FS protocol of File Sharing. Keeping my head down and trying it out...

I posted a PR with some changes that I made to get Tigge’s openant library to work. Basically, I put a pause after the reset line that you mentioned above and bypassed the use of udev_rules as it doesn’t apply in Windows. You can use libusb but installation is a bit different. I’ve added Windows installation instructions to the readme in the PR with details on what worked for me.

Logging inside Threads When Logger is Already Configured

EDIT: Repo with all code (branch "daemon"). The question is regarding the code in the file linked to).
My main program configures logging like this (options have been simplified):
logging.basicConfig(level='DEBUG', filename="/some/directory/cstash.log")
Part of my application starts a daemon, for which I use the daemon package:
with daemon.DaemonContext(
pidfile=daemon.pidfile.PIDLockFile(self.pid_file),
stderr=self.log_file,
stdout=self.log_file
):
self.watch_files()
where self.log_file is a file I've opened for writing.
When I start the application, I get:
--- Logging error ---
Traceback (most recent call last):
File "/Users/afraz/.pyenv/versions/3.7.2/lib/python3.7/logging/__init__.py", line 1038, in emit
self.flush()
File "/Users/afraz/.pyenv/versions/3.7.2/lib/python3.7/logging/__init__.py", line 1018, in flush
self.stream.flush()
OSError: [Errno 9] Bad file descriptor
If I switch off the logging to a file in the daemon, the logging in my main application works, and if I turn off the logging to a file in my main application, the logging in the daemon works. If I set them up to log to a file (even different files), I get the error above.

After trying many things, here's what worked:
def process_wrapper():
with self.click_context:
self.process_queue()
def watch_wrapper():
with self.click_context:
self.watch_files()
with daemon.DaemonContext(
pidfile=daemon.pidfile.PIDLockFile(self.pid_file),
files_preserve=[logger.handlers[0].stream.fileno()],
stderr=self.log_file,
stdout=self.log_file
):
logging.info("Started cstash daemon")
while True:
threading.Thread(target=process_wrapper).start()
time.sleep(5)
threading.Thread(target=watch_wrapper).start()
There were two main things wrong:
daemon.DaemonContext needs files_preserve set to the file logging handler, so it doesn't close the file once the context is switched. This is the actual solution to the original problem.
Further however, both methods needed to be in separate threads, not just one. The while True loop in the main thread was stopping the other method from running, so putting them both into separate threads means they can both run

Boto3/Jenkins client throwing an error while running the code

I am running a daily glue script in one of our AWS machines, which I scheduled it using jenkins.
I am getting the following from the last 15 days. (this daily job is running for almost 6 months and all of a sudden since the 15 days this is happening)
The jenkins console output looks like this
Started by timer
Building in workspace /var/lib/jenkins/workspace/build_name_xyz
[build_name_xyz] $ /bin/sh -xe /tmp/jenkins8188702635955396537.sh
+ /usr/bin/python3 /var/lib/jenkins/path_to_script/glue_crawler.py
Traceback (most recent call last):
File "/var/lib/jenkins/path_to_script/glue_crawler.py", line 10, in <module>
response = glue_client.update_crawler(Name = crawler_name,Targets = {'S3Targets': [{'Path':update_path}]})
File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidInputException: An error occurred (InvalidInputException) when calling the UpdateCrawler operation: Cannot update Crawler while running. Please stop crawl or wait until it completes to update.
Build step 'Execute shell' marked build as failure
Finished: FAILURE
So, I went ahead and have seen the line 10 in this file
/var/lib/jenkins/path_to_script/glue_crawler.py
That looked something like this.
import boto3
import datetime
glue_client = boto3.client('glue', region_name='region_name')
crawler_name = 'xyz_abc'
today = (datetime.datetime.now()).strftime("%Y_%m_%d")
update_path = 's3://path-to-respective-aws-s3-bucket/%s' % (today)
response = glue_client.update_crawler(Name = crawler_name,Targets = {'S3Targets': [{'Path':update_path}]})
response_crawler = glue_client.start_crawler(
Name=crawler_name
)
print(response_crawler)
The above throws an error at line 10. I am not understanding what exactly is going wrong on line 10 and hence the jenkins throws an error with the red ball, requesting for some help here. I tried googling on this, but I couldn't find anything.
Just, FYI......if I run the same build (by clicking 'Build Now') using the jenkins UI after sometime, the job runs absolutely fine.
Not sure what exactly is wrong here, any help is highly appreciated.
Thanks in advance!!

The error is self explanatory:
Cannot update Crawler while running. Please stop crawl or wait until it completes to update.
So somehow the crawler was started approximately at the same time and in Glue it's not allowed to update crawler properties when it's running. Please check if there is any other task that starts crawler with name xyz_abc too. Besides that in AWS Console make sure the crawler is configured to run on demand rather than on schedule.

Exceptional calls to urllib.request.urlopen when called from ThreadPoolExecutor leaves open file descriptors

I'm trying to download a lot of data using multiple threads from Yahoo Finance. I'm using concurrent.futures.ThreadPoolExecutor to speed things up. Everything goes well until I consume all the available file descriptors (1024 by default).
When urllib.request.urlopen() raises an exception the file descriptor is left open (no matter what timeout for socket I use). Normally this file descriptor is reused if I run stuff only from a single (main) thread so this problem doesn't occur. But when these exceptional urlopen() calls are made from ThreadPoolExecutor threads these file descriptors are left open. The only solution I have come up with so far is to use either processes (ProcessPoolExecutor) which is very cumbersome and inefficient or increase the number of allowed file descriptors to something really big (not all the potential users of my library are going to do this anyway). There must be a smarter way to deal with this problem.
And also I wonder whether this is a bug in Python libraries or am I just doing something wrong...
I'm running Python 3.4.1 on Debian (testing, kernel 3.10-3-amd64).
This is an example code that demonstrates this behaviour:
import concurrent
import concurrent.futures
import urllib.request
import os
import psutil
from time import sleep
def fetchfun(url):
urllib.request.urlopen(url)
def main():
print(os.getpid())
p = psutil.Process(os.getpid())
print(p.get_num_fds())
# this url doesn't exist
test_url = 'http://ichart.finance.yahoo.com/table.csv?s=YHOOxyz' + \
'&a=00&b=01&c=1900&d=11&e=31&f=2019&g=d'
with concurrent.futures.ThreadPoolExecutor(1) as executor:
futures = []
for i in range(100):
futures.append(executor.submit(fetchfun, test_url))
count = 0
for future in concurrent.futures.as_completed(futures):
count += 1
print("{}: {} (ex: {})".format(count, p.get_num_fds(), future.exception()))
print(os.getpid())
sleep(60)
if __name__ == "__main__":
main()

When the HTTPError is raised, it saves a reference to the HTTPResponse object for the request as the fp attribute of the HTTPError. That reference gets saved in your futures list, which isn't destroyed until your program ends. That means there's a reference to the HTTPResponse being kept alive for your entire program. As long as that reference exists, the socket used in the HTTPResponse stays open. One way you can work around this is by explicitly closing the HTTPResponse when you handle the exception:
with concurrent.futures.ThreadPoolExecutor(1) as executor:
futures = []
for i in range(100):
futures.append(executor.submit(fetchfun, test_url))
count = 0
for future in concurrent.futures.as_completed(futures):
count += 1
exc = future.exception()
print("{}: {} (ex: {})".format(count, p.get_num_fds(), exc))
exc.fp.close() # Close the HTTPResponse

ZEO ZODB database - run locally not working

I tried looking at the documentation for running ZEO on a ZODB database, but it isn't working how they say it should.
I can get a regular ZODB running fine, but I would like to make the database accessible by several processes for a program, so I am trying to get ZEO to work.
I created this script in a folder with a subfolder zeo, which will hold the "database.fs" files created by the make_server function in a different parallel process:
CODE:
from ZEO import ClientStorage
import ZODB
import ZODB.config
import os, time, site, subprocess, multiprocessing
# make the server in for the database in a separate process with windows command
def make_server():
runzeo_path = site.getsitepackages()[0] + "\Lib\site-packages\zeo-4.0.0-py2.7.egg\ZEO\\runzeo.py"
filestorage_path = os.getcwd() + '\zeo\database.fs'
subprocess.call(["python", runzeo_path, "-a", "127.0.0.1:9100", "-f" , filestorage_path])
if __name__ == "__main__":
server_process = multiprocessing.Process(target = make_server)
server_process.start()
time.sleep(5)
storage = ClientStorage.ClientStorage(('localhost', 9100), wait=False)
db = ZODB.DB(storage)
connection = db.open()
root = connection.root()
the program will just block at the ClientStorage line if the wait=False is not given.
If the wait=False is given it produces this error:
Error Message:
Traceback (most recent call last):
File "C:\Users\cbrown\Google Drive\EclipseWorkspace\NewSpectro - v1\20131202\2 - database\zeo.py", line 17, in <module>
db = ZODB.DB(storage)
File "C:\Python27\lib\site-packages\zodb-4.0.0-py2.7.egg\ZODB\DB.py", line 443, in __init__
temp_storage.load(z64, '')
File "C:\Python27\lib\site-packages\zeo-4.0.0-py2.7.egg\ZEO\ClientStorage.py", line 841, in load
data, tid = self._server.loadEx(oid)
File "C:\Python27\lib\site-packages\zeo-4.0.0-py2.7.egg\ZEO\ClientStorage.py", line 88, in __getattr__
raise ClientDisconnected()
ClientDisconnected
Here is the output from the cmd prompt for my process which runs a server:
------
2013-12-06T21:07:27 INFO ZEO.runzeo (7460) opening storage '1' using FileStorage
------
2013-12-06T21:07:27 WARNING ZODB.FileStorage Ignoring index for C:\Users\cab0008
\Google Drive\EclipseWorkspace\NewSpectro - v1\20131202\2 - database\zeo\databas
e.fs
------
2013-12-06T21:07:27 INFO ZEO.StorageServer StorageServer created RW with storage
s: 1:RW:C:\Users\cab0008\Google Drive\EclipseWorkspace\NewSpectro - v1\20131202\
2 - database\zeo\database.fs
------
2013-12-06T21:07:27 INFO ZEO.zrpc (7460) listening on ('127.0.0.1', 9100)
What could I be doing wrong? I just want this to work locally right now so there shouldn't be any need for fancy web stuff.

You should use proper process management and simplify your life. You likely want to look into supervisor, which can be responsible for running/starting/stopping your application and ZEO.
Otherwise, you need to look at the double-fork trick to daemonize ZEO -- but why bother when a process management tool like supervisor does this for you.
If you are savvy with relational database administration, and already have a relational database at your disposal -- you can also consider RelStorage as a very good ZODB (low-level) storage backend.

In Windows you should use double \ instead of a single \ in the paths. Easy and portable way to accomplish this is to use os.path.join() function, eg. os.path.join('os.getcwd()', 'zeo', 'database.fs'). Otherwise a similar code worked ok for me.

Had same error on Windows , on Linux everything OK ...
your code is ok , to make this to work change following
C:\Python33\Lib\site-packages\ZEO-4.0.0-py3.3.egg\ZEO\zrpc\trigger.py ln:235
self.trigger.send(b'x')
C:\Python33\Lib\site-packages\ZEO-4.0.0-py3.3.egg\ZEO\zrpc\client.py ln:458:459 - comment them
here is those lines:
if socktype != socket.SOCK_STREAM:
continue