Develop Reference

Python access FTDI device by pyFTDI but get the INVALID_HANDLE error

I want to establish a Python platform so that I can read and write my FTDI device
(USB-Blaster, Altera DE0-nano).
And fortunately, I got a great example on Github called ftdi2.py, it can detect my USB-Blaster precisely, but when I try to do something on this device such as Reset or Read/Write byte array, it always shows ftdi2.FTDeviceError: 'INVALID_HANDLE'
I couldn't find out what was wrong with this code. May it is the wrong data type of ft.HANDLE or something else in the class "SLD_Controller"?
from __future__ import print_function
import ctypes as c
import subprocess
from time import sleep
from bitstring import BitArray
from sys import platform as _platform
from ftdi2 import *
#------------------------------------------------------------------------------
#
# Create a write buffer as a ctypes array of bytes
def tx_buffer(byte_list):
return (c.c_ubyte * len(byte_list))(*byte_list)
#-------------------------------------------------------------------------------
#
# FT245 Bit definitions for JTAG mode
OFF = 0x00
TCK = 0x01
TMS = 0x02
TDI = 0x10
LED = 0x20
RD = 0x40
sHM = 0x80
# Bit-mode - Two byte codes
M0D0R = [LED | RD , LED | TCK]
M0D1R = [LED | TDI | RD , LED | TDI | TCK]
M1D0R = [LED | TMS | RD , LED | TMS | TCK]
M1D1R = [LED | TMS | TDI | RD , LED | TMS | TDI | TCK]
M0D0 = [LED , LED | TCK]
M0D1 = [LED | TDI , LED | TDI | TCK]
M1D0 = [LED | TMS , LED | TMS | TCK]
M1D1 = [LED | TMS | TDI , LED | TMS | TDI | TCK]
# TAP controller Reset
TAP_RESET = tx_buffer(M1D0 + M1D0 + M1D0 + M1D0 + M1D0)
# TAP controller Reset to Idle
TAP_IDLE = tx_buffer(M0D0)
#-------------------------------------------------------------------------------
#
# A class for higher level SLD functions
#
# instruction/data argument is a BitArray instance
#
# All start and end in the Run_Test/Idle TAP state
class SLD_Controller(object):
def __init__(self, interface_name, m_width, n_width, csv_file_name = ''):
self.interface = open_ex_by_name(interface_name)
self.instruction_width = 10
self.virtual_inst_width = m_width
self.node_adrs_width = n_width
self.interface.reset_device()
self.interface.write(TAP_RESET)
self.interface.write(TAP_IDLE)
#----
def TAP_Reset(self):
self.interface.write(TAP_RESET)
self.interface.write(TAP_IDLE)
#----
def close(self):
self.interface.close()
#===============================================================================
#
# Main
print('Testing')
dev = list_devices()
print('devices:', dev)
sld = SLD_Controller(b'USB-Blaster', 4, 1)
sld.TAP_Reset()
sld.close()
print()
print('closed')
print('Done')

Traceroute - Sometimes the sockets fail

I'm writing traceroute in Python3 as an exercise. It's working quite well, but every now and then, the trace will be unable to receive an answer from a certain hop and keep incrementing the TTL without ever getting a response. The regular linux traceroute program does not suffer from this problem so it has to be something in my code.
I've tried different variations, but they all suffer from the same problem. I'll run the trace on 8.8.8.8 and 4 out of 5 times, it works perfectly.
I've tried creating sockets using the context manager (with socket as ...) and also manually.
import os
import random
import socket
import sys
import time
class Traceroute:
def __init__(self, dest, maxhops=30, port=random.randrange(33434,33534), host=''):
self.dest = dest
self.maxhops = maxhops
self.port = port
self.host = host
self.hops = []
# Function for creating new sockets
def get_socket(self,type,proto=0):
s = socket.socket(
family=socket.AF_INET,
type=type,
proto=proto
)
s.bind((self.host,self.port))
return s
# Do the trace and store hop information for later use
def get_path(self):
try:
self.dest_ip = socket.gethostbyname(self.dest)
self.dest_name = socket.gethostbyaddr(self.dest)[0]
except socket.gaierror as e:
print(f'Error: {self.dest} - {e}')
address = ['']
ttl = 1
while ttl < self.maxhops +1 and address[0] != self.dest_ip:
receiver_socket = self.get_socket(socket.SOCK_RAW,socket.getprotobyname('icmp'))
receiver_socket.settimeout(2)
sender_socket = self.get_socket(socket.SOCK_DGRAM,socket.getprotobyname('udp'))
sender_socket.setsockopt(socket.SOL_IP,socket.IP_TTL,ttl)
sender_socket.sendto(b'',(self.dest_ip,self.port))
tries = 3
while tries > 0:
starttime = time.time()
try:
data, address = receiver_socket.recvfrom(1024)
latency = [round((time.time()-starttime)*1000,2)]
try:
hostname = socket.gethostbyaddr(address[0])[0]
except socket.herror as e:
hostname = address[0]
self.hops.append({
'address':address[0],
'hostname':hostname,
'latency': latency
})
tries = 0
except socket.error:
tries -= 1
if tries == 0:
self.hops.append({
'address':None,
'hostname':ttl,
'latency':None
})
finally:
receiver_socket.close()
sender_socket.close()
ttl += 1
os.system('clear')
self.draw()
def draw(self):
name = f'{self.dest_ip} - {self.dest_name}' if self.dest_name != self.dest_ip else self.dest_ip
print(f'Traceroute to {name}')
# Print the full hop list, for debugging
for hop in self.hops:
print(hop)
# Print the hops in a readable fashion
for hop in self.hops:
if hop['address'] is None:
print(f"{self.hops.index(hop)+1} - {'*':<50} - {'*':<50} - {'*':<50}")
else:
latencylen = len(hop['latency'])
latencyavg = str(round(sum(hop['latency'])/latencylen,2)) + 'ms'
name = f'{hop["address"]} - {hop["hostname"]}' if hop['address'] != hop['hostname'] else hop['address']
print(f'{self.hops.index(hop)+1} - {name:<50} - {latencyavg:<50} - LEN:{latencylen:<50}')
def main():
if len(sys.argv) < 2:
print('Please enter a destination')
else:
trace = Traceroute(sys.argv[1])
trace.get_path()
if __name__ == '__main__':
main()
Here is the output when it completely bugs out:
self.hops is a list of dictionaries.
The stuff in {brackets} is the hop dictionary itself. I added it for debugging.
Note that hops 2 and 3 are identical. So the trace somehow ran against this hop twice. The number at the beginning of the line is the list index, which also represents the TTL. I usually reach 8.8.8.8 by the 9th hop, but in this case it just keeps going, incrementing the TTL until it reaches the maximum.
Traceroute to 8.8.8.8 - google-public-dns-a.google.com
{'address': '192.168.1.1', 'hostname': '_gateway', 'latency': [8.21]}
{'address': '104.195.128.122', 'hostname': '104-195-128-122.cpe.teksavvy.com', 'latency': [1572.38]}
{'address': '104.195.128.122', 'hostname': '104-195-128-122.cpe.teksavvy.com', 'latency': [15.97]}
{'address': '104.195.129.9', 'hostname': '104-195-129-9.cpe.teksavvy.com', 'latency': [27.22]}
{'address': '206.248.155.92', 'hostname': 'ae10-0-bdr01-tor2.teksavvy.com', 'latency': [20.05]}
{'address': '206.248.155.10', 'hostname': 'ae12-0-bdr01-tor.teksavvy.com', 'latency': [27.2]}
{'address': None, 'hostname': 7, 'latency': None}
{'address': None, 'hostname': 8, 'latency': None}
{'address': None, 'hostname': 9, 'latency': None}
{'address': None, 'hostname': 10, 'latency': None}
{'address': None, 'hostname': 11, 'latency': None}
{'address': None, 'hostname': 12, 'latency': None}
{'address': None, 'hostname': 13, 'latency': None}
{'address': None, 'hostname': 14, 'latency': None}
{'address': None, 'hostname': 15, 'latency': None}
1 - 192.168.1.1 - _gateway - 8.21ms - LEN:1
2 - 104.195.128.122 - 104-195-128-122.cpe.teksavvy.com - 1572.38ms - LEN:1
3 - 104.195.128.122 - 104-195-128-122.cpe.teksavvy.com - 15.97ms - LEN:1
4 - 104.195.129.9 - 104-195-129-9.cpe.teksavvy.com - 27.22ms - LEN:1
5 - 206.248.155.92 - ae10-0-bdr01-tor2.teksavvy.com - 20.05ms - LEN:1
6 - 206.248.155.10 - ae12-0-bdr01-tor.teksavvy.com - 27.2ms - LEN:1
7 - * - * - *
8 - * - * - *
9 - * - * - *
10 - * - * - *
11 - * - * - *
12 - * - * - *
13 - * - * - *
14 - * - * - *
15 - * - * - *
Thanks,

Creating a formatted table from a list of tuples

I have a list of data that looks something like this:
[
(
1,
u'python -c \'print("ok")\'',
u'data',
u'python'
), (
2,
u'python -c \'print("this is some data")\'',
u'data',
u'python'
)
]
This data is taken out of a database and displayed as this, and is continuously growing. What I would like to do is display the data like this:
Language | Type | Payload
-------------------------------
python | data | python -c 'print("ok")'
python | data | python -c 'print("this is some data")'
I have a function that kinda does the same thing, but it's not exactly as expected:
def print_table(data, cols, width):
n, r = divmod(len(data), cols)
pattern = "{{:{}}}".format(width)
line = "\n".join(pattern * cols for _ in range(n))
last_line = pattern * r
print(line.format(*data))
print(last_line.format(*data[n*cols]))
How can I get the output of my data to look as wanted? From the answers it's possible with pandas but I would also like a way to do it without installing external modules

Analyze the data for its max-width and use string formatting - some 'creative' formatting later:
data = [
(
1,
u'python -c \'print("ok")\'',
u'data',
u'python'
), (
2,
u'python -c \'print("this is some data")\'',
u'data',
u'python'
)
]
def print_table(data):
widths = {0:0, 3:len("Language"),2:len("Type"),1:len("Payload")}
for k in data:
for i,d in enumerate(k):
widths[i] = max(widths[i],len(str(d)))
# print(widths)
lan, typ, pay = ["Language","Type","Payload"]
print(f"{lan:<{widths[3]}} | {typ:<{widths[2]}} | {pay:<{widths[1]}}")
# adjust by 10 for ' | ' twice
print("-" * (widths[1]+widths[2]+widths[3]+10))
for k in data:
_, pay, typ, lan = k
print(f"{lan:<{widths[3]}} | {typ:<{widths[2]}} | {pay:<{widths[1]}}")
Output:
Language | Type | Payload
------------------------------------------------------------
python | data | python -c 'print("ok")'
python | data | python -c 'print("this is some data")'
string format mini language
Equivalent Python 2.7 code:
# w == widths - would break 79 chars/line else wise
def print_table(data):
w = {0:0, 3:len("Language"),2:len("Type"),1:len("Payload")}
for k in data:
for i,d in enumerate(k):
w[i] = max(w[i],len(str(d)))
lan, typ, pay = ["Language","Type","Payload"]
print "{:<{}} | {:<{}} | {:<{}}".format(lan, w[3], typ, w[2], pay, w[1])
print "-" * (w[1]+w[2]+w[3]+10)
for k in data:
_, pay, typ, lan = k
print "{:<{}} | {:<{}} | {:<{}}".format(lan, w[3], typ, w[2], pay, w[1])

Solution that handles any number of columns:
from operator import itemgetter
data = [
('ID', 'Payload', 'Type', 'Language'),
(1, u'python -c \'print("ok")\'', u'data', u'python'),
(2, u'python -c \'print("this is some data")\'', u'data', u'python')
]
def print_table(data):
lengths = [
[len(str(x)) for x in row]
for row in data
]
max_lengths = [
max(map(itemgetter(x), lengths))
for x in range(0, len(data[0]))
]
format_str = ''.join(map(lambda x: '%%-%ss | ' % x, max_lengths))
print(format_str % data[0])
print('-' * (sum(max_lengths) + len(max_lengths) * 3 - 1))
for x in data[1:]:
print(format_str % x)
print_table(data)
Output:
$ python table.py
ID | Payload | Type | Language |
---------------------------------------------------------------
1 | python -c 'print("ok")' | data | python |
2 | python -c 'print("this is some data")' | data | python |

You can use pandas for that:
import pandas as pd
data = pd.DataFrame(a, columns=['id','Payload', 'type', 'Language'])
print(data)
gives you:
id Payload type Language
0 1 python -c 'print("ok")' data python
1 2 python -c 'print("this is some data")' data python

Does pytest support unicode in test name?

Here is my test sample (test_time.py):
#!/usr/bin/python
# -*- coding: utf-8 -*-
import pytest
from datetime import datetime, timedelta
testdata = [
(datetime(2001, 12, 12), datetime(2001, 12, 11), timedelta(1)),
(datetime(2001, 12, 11), datetime(2001, 12, 12), timedelta(-1)),
]
#pytest.mark.parametrize("a,b,expected", testdata, ids=[u"中文", u"English"])
def test_timedistance_v1(a, b, expected):
diff = a - b
assert diff != expected
Here is the pytest output:
============================================================================== FAILURES ==============================================================================
_________________________________________________________________ test_timedistance_v1[\u4e2d\u6587] _________________________________________________________________
a = datetime.datetime(2001, 12, 12, 0, 0), b = datetime.datetime(2001, 12, 11, 0, 0), expected = datetime.timedelta(1)
#pytest.mark.parametrize("a,b,expected", testdata, ids=[u"中文", u"English"])
def test_timedistance_v1(a, b, expected):
diff = a - b
> assert diff != expected
E assert datetime.timedelta(1) != datetime.timedelta(1)
test_time.py:15: AssertionError
___________________________________________________________________ test_timedistance_v1[English] ____________________________________________________________________
a = datetime.datetime(2001, 12, 11, 0, 0), b = datetime.datetime(2001, 12, 12, 0, 0), expected = datetime.timedelta(-1)
#pytest.mark.parametrize("a,b,expected", testdata, ids=[u"中文", u"English"])
def test_timedistance_v1(a, b, expected):
diff = a - b
> assert diff != expected
E assert datetime.timedelta(-1) != datetime.timedelta(-1)
test_time.py:15: AssertionError
====================================================================== 2 failed in 0.05 seconds ======================================================================
For the second line in output , the test name is "test_timedistance_v1[\u4e2d\u6587]" , I hope it's "test_timedistance_v1[中文]"， does py.test support it?
(my pytest version is 3.1.2, OS: macOS 10.12.5)

It does not depend of pytest but of your computer locale.
Here the trace-log of test (LC_ALL="en_US.UTF-8") :
================================ test session starts ================================
platform linux -- Python 3.5.3, pytest-2.9.2, py-1.4.34, pluggy-0.3.1
rootdir: /home/..../tmp, inifile:
collected 2 items
pytest_chin.py FF
===================================== FAILURES ======================================
_____________________________ test_timedistance_v1[中文] ______________________________
...
And with with LC_ALL="fr_FR.iso8859-1" :
================================ test session starts ================================
platform linux -- Python 3.5.3, pytest-2.9.2, py-1.4.34, pluggy-0.3.1
rootdir: /home/gustavi/tmp, inifile:
collected 2 items
pytest_chin.py FF
===================================== FAILURES ======================================
\x1b[1m\x1b[31m_____________________________ test_timedistance_v1[\u4e2d\u6587] ______________________________\x1b[0m
...
Here an usefull link to setup your locale on OS X.

libvirt: From state integer to string?

I have this simple script to check the memory usage of virtual machines managed by libvirt.
How can I convert the integer for state from dom.info() to a human readable string?
import libvirt
import re
import sys
def mem_total_kb():
meminfo = open('/proc/meminfo').read()
matched = re.search(r'^MemTotal:\s+(\d+)', meminfo)
return int(matched.groups()[0])
def main():
conn = libvirt.openReadOnly(None)
if conn == None:
print 'Failed to open connection to the hypervisor'
sys.exit(1)
used_mem_sum = 0
for domain_id in conn.listDomainsID():
dom = conn.lookupByID(domain_id)
state, max_mem, used_mem, vcpus, cpu_time_used = dom.info()
print(
'name=%s state=%s max_mem=%s used_mem=%s vcpus=%s cpu_time_used=%s' % (dom.name(), state, max_mem, used_mem, vcpus, cpu_time_used))
used_mem_sum += used_mem
print('Sum of used mem: %s KiB' % used_mem_sum)
mem_total = mem_total_kb()
print('Sum of physical mem: %s KiB' % mem_total)
if used_mem_sum > mem_total:
print('########## VMs use more RAM than available!')
return
mem_left=mem_total - used_mem_sum
print('Memory left: %s KiB' % (mem_left))
mem_left_should=4000000
if mem_left<mem_left_should:
print('less than mem_left_should=%sKiB left!' % mem_left_should)
if __name__ == '__main__':
main()
Docs: https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainInfo
state: the running state, one of virDomainState
enum virDomainState {
VIR_DOMAIN_NOSTATE = 0
no state
VIR_DOMAIN_RUNNING = 1
the domain is running
VIR_DOMAIN_BLOCKED = 2
the domain is blocked on resource
VIR_DOMAIN_PAUSED = 3
the domain is paused by user
VIR_DOMAIN_SHUTDOWN = 4
the domain is being shut down
VIR_DOMAIN_SHUTOFF = 5
the domain is shut off
VIR_DOMAIN_CRASHED = 6
the domain is crashed
VIR_DOMAIN_PMSUSPENDED = 7
the domain is suspended by guest power management
VIR_DOMAIN_LAST = 8
NB: this enum value will increase over time as new events are added to the libvirt API. It reflects the last state supported by this version of the libvirt API.
}

Looks like at least the enum constant names are exposed in the libvirt module. The following works for me...
import libvirt
import pprint
import re
d = {}
for k, v in libvirt.__dict__.iteritems():
if re.match('VIR_DOMAIN_[A-Z]+$', k):
d[v] = k
pprint.pprint(d)
...and prints...
{0: 'VIR_DOMAIN_NOSTATE',
1: 'VIR_DOMAIN_RUNNING',
2: 'VIR_DOMAIN_BLOCKED',
3: 'VIR_DOMAIN_PAUSED',
4: 'VIR_DOMAIN_SHUTDOWN',
5: 'VIR_DOMAIN_SHUTOFF',
6: 'VIR_DOMAIN_CRASHED',
7: 'VIR_DOMAIN_PMSUSPENDED'}
The descriptions, which are likely just comments in the original source code, are probably not exposed. One of the examples defines its own dictionary on line 106...
state_names = { libvirt.VIR_DOMAIN_RUNNING : "running",
libvirt.VIR_DOMAIN_BLOCKED : "idle",
libvirt.VIR_DOMAIN_PAUSED : "paused",
libvirt.VIR_DOMAIN_SHUTDOWN : "in shutdown",
libvirt.VIR_DOMAIN_SHUTOFF : "shut off",
libvirt.VIR_DOMAIN_CRASHED : "crashed",
libvirt.VIR_DOMAIN_NOSTATE : "no state" }
...so you could just take it from there, although it seems to be incomplete.

This bash script will fetch the list of virDomainState values (excluding VIR_DOMAIN_LAST) and their descriptions in JSON format:
#!/bin/bash
HEADER=include/libvirt/libvirt-domain.h
get_header_file()
{
curl -s https://raw.githubusercontent.com/libvirt/libvirt/master/"$HEADER"
}
select_virDomainState_block()
{
sed -n '/virDomainState:/,/^\} virDomainState;/ p'
}
join_multiline_comments()
{
sed -n '\%/\*[^*]*$% N; \%/\*.*\*/$% { s/\s*\n\s*/ /; p; }'
}
enum_values_to_json_map()
{
echo '{'
sed "s/\s*VIR_DOMAIN\S\+\s*=\s*//; s^,\s*/\*\s*^ : '^; s^\s*\*/^',^;"
echo '}'
}
get_header_file \
| select_virDomainState_block \
| join_multiline_comments \
| grep 'VIR_DOMAIN\S\+\s*=' \
| enum_values_to_json_map
Example usage:
$ ./get_libvirt_domain_states
{
0 : 'no state',
1 : 'the domain is running',
2 : 'the domain is blocked on resource',
3 : 'the domain is paused by user',
4 : 'the domain is being shut down',
5 : 'the domain is shut off',
6 : 'the domain is crashed',
7 : 'the domain is suspended by guest power management',
}
Note that the script downloads a ~150KB file from the libvirt mirror repository at GitHub. It is intended for facilitating staying up-to-date with the libvirt code. Of course you can call it from your python code, but personally I wouldn't do that.