Develop Reference

Cloning volume using Libvirt Python createXMLFrom causes End of file while reading data: Input/output error

I am trying to clone a volume using:
libvirt_conn = libvirt.openAuth(<qemu_addr>, <auth>, 0)
pool = libvirt_conn.storagePoolLookupByName(<pool_name>)
original_volume = pool.storageVolLookupByName(<original_volume_name>)
new_volume_xml = <xml_string>
new_volume = pool.createXMLFrom(new_volume_xml, original_volume)
When I run this I get the following error:
End of file while reading data: Input/output error
When I try:
libvirt_conn = libvirt.openAuth(<qemu_addr>, <auth>, 0)
pool = libvirt_conn.storagePoolLookupByName(<pool_name>)
original_volume = pool.storageVolLookupByName(<original_volume_name>)
new_volume_xml = <xml_string>
try:
new_volume = pool.createXMLFrom(new_volume_xml, original_volume)
except:
<next libvirt command>
I get a client socket is closed error. I have tried editing /etc/libvirt/libvirtd.conf with:
min_workers = 5
max_workers = 20
log_level = 1
log_filters="1:libvirt 1:util 1:qemu"
log_outputs="1:file:/var/log/libvirt/libvirtd.log"
keepalive_interval = -1
When I restart libvirtd and tail /var/log/libvirt/libvirtd.log I don't see anything useful. My feeling is the socket is closing because the cloning of the volume takes a longtime, but I am not sure how to keep the libvirt/qemu socket open longer. Is this possible?

Adding more advanced assertion to Python unittest

I'm testing Trello API, creating and deleting cards. The only assertions I have here are for status code. How can I add some more 'advanced' assertions to the below code?
import unittest
from post_and_delete import *
class TestBasic(unittest.TestCase):
def test_post_delete(self):
# create new card with random name
name = nonce(10)
result_post = post(name)
self.assertEqual(result_post.json()['name'], name)
self.assertEqual(result_post.status_code, 200)
card_id = result_post.json()['id']
# get the card, verify it exists, status code should be 200
result_get = get(card_id)
self.assertEqual(result_get.status_code, 200)
# delete the card, check again if status code is 200
result = delete(card_id)
self.assertEqual(result.status_code, 200)
# get the recently deleted card, status code should be 404
result_get = get(card_id)
self.assertEqual(result_get.status_code, 404)
# try to delete the card again, status code should be 404
result = delete(card_id)
self.assertEqual(result.status_code, 404)
if __name__ == '__main__':
unittest.main()

MinimalModbus read_registry error

I am trying to read registry information from a modbus device using MinimalModbus. However, every time I attempt to read registry 40003 which has a value of 220 I receive this error:
raise ValueError('The slave is indicating an error. The response is: {!r}'.format(response))
ValueError: The slave is indicating an error. The response is: '\x01\x83\x02Àñ'
I know there is a value in 40003 and I am following the communication documents for the device. Here is my code:
import minimalmodbus
import serial
gas = minimalmodbus.Instrument('COM5', 1)
gas.serial.baudrate = 9600
gas.serial.bytesize = 8
gas.serial.parity = serial.PARITY_NONE
gas.serial.stopbits = 1
gas.serial.timeout = 0.05
gas.mode = minimalmodbus.MODE_RTU
temp = gas.read_register(40003, 1)
print (float(temp))
I have this problem for every registry and I cannot find information regarding Àñ.

The problem was the registry number 40003. I guess the modbus protocol doesn't require the full registry number, so I changed it to temp = gas.read_register(3, 1)

Streamparse/Python - custom fail() method not working for error tuples

I'm using Storm to process messages off of Kafka in real-time and using streamparse to build my topology. For this use case, it's imperative that we have 100% guarantee that any message into Storm is processed and ack'd. I have implemented logic on my bolt using try/catch (see below), and I would like to have Storm replay these messages in addition to writing this to another "error" topic in Kafka.
In my KafkaSpout, I assigned the tup_id to equal the offset id from the Kafka topic that my consumer is feeding from. However, when I force an error in my Bolt using a bad variable reference, I'm not seeing the message be replayed. I am indeed seeing one write to the 'error' Kafka topic, but only once--meaning that the tuple is never being resubmitted into my bolt(s). My setting for the TOPOLOGY_MESSAGE_TIMEOUT_SEC=60 and I'm expecting Storm to keep replaying the failed message once every 60 seconds and have my error catch keep writing to the error topic, perpetually.
KafkaSpout.py
class kafkaSpout(Spout):
def initialize(self, stormconf, context):
self.kafka = KafkaClient(str("host:6667"))#,offsets_channel_socket_timeout_ms=60000)
self.topic = self.kafka.topics[str("topic-1")]
self.consumer = self.topic.get_balanced_consumer(consumer_group=str("consumergroup"),auto_commit_enable=False,zookeeper_connect=str("host:2181"))
def next_tuple(self):
for message in self.consumer:
self.emit([json.loads(message.value)],tup_id=message.offset)
self.log("spout emitting tuple ID (offset): "+str(message.offset))
self.consumer.commit_offsets()
def fail(self, tup_id):
self.log("failing logic for consumer. resubmitting tup id: ",str(tup_id))
self.emit([json.loads(message.value)],tup_id=message.offset)
processBolt.py
class processBolt(Bolt):
auto_ack = False
auto_fail = False
def initialize(self, conf, ctx):
self.counts = Counter()
self.kafka = KafkaClient(str("host:6667"),offsets_channel_socket_timeout_ms=60000)
self.topic = self.kafka.topics[str("topic-2")]
self.producer = self.topic.get_producer()
self.failKafka = KafkaClient(str("host:6667"),offsets_channel_socket_timeout_ms=60000)
self.failTopic = self.failKafka.topics[str("topic-error")]
self.failProducer = self.failTopic.get_producer()
def process(self, tup):
try:
self.log("found tup.")
docId = tup.values[0]
url = "solrserver.host.com/?id="+str(docId)
thisIsMyForcedError = failingThisOnPurpose ####### this is what im using to fail my bolt consistent
data = json.loads(requests.get(url).text)
if len(data['response']['docs']) > 0:
self.producer.produce(json.dumps(docId))
self.log("record FOUND {0}.".format(docId))
else:
self.log('record NOT found {0}.'.format(docId))
self.ack(tup)
except:
docId = tup.values[0]
self.failProducer.produce( json.dumps(docId), partition_key=str("ERROR"))
self.log("TUP FAILED IN PROCESS BOLT: "+str(docId))
self.fail(tup)
I would appreciate any help with how to correctly implement the custom fail logic for this case. Thanks in advance.

IB/IbPy-Understanding how to access and store a variable from API response

My understanding of Interactive Brokers API is of an async nature where if I call reqMktData() after connecting, it calls tickPrice() among other methods, sends parameters to tickPrice(), then tickPrice() passes it's own results to a message object. To read the incoming message objects, I need to implement a message handler from EWrapper.
I am trying to extract the price variable inside the message object, but I have been unsuccessful in extracting the tickPrice() price field or extracting the price inside the message object either from calling tickPrice() directly or from something like msg.price.
Here is my code:
from ib.opt import ibConnection, Connection, message
import time
import ib_data_types as datatype
from ib.ext.Contract import Contract
#price_field = ''
def reply_handler(msg):
print("Reply:", msg)
#def my_callback_handler(msg):
#if msg.field == 4:
#price_field = msg.price
def request_streaming_data(ib_conn):
# Stream market data
ib_conn.reqMktData(1,
contract,
datatype.GENERIC_TICKS_NONE,
datatype.SNAPSHOT_NONE)
time.sleep(1)
if __name__ == "__main__":
tws_conn = ibConnection(host='localhost', port=7497, clientId=100)
tws_conn.connect()
tws_conn.registerAll(reply_handler)
#tws_conn.register(my_callback_handler, "TickPrice")
contract = Contract()
contract.m_symbol = 'GE'
contract.m_exchange = 'SMART'
contract.m_currency = 'USD'
contract.m_secType = 'STK'
request_streaming_data(tws_conn)
#print(price_field)
The above code works perfectly fine for printing to screen the responses from API in the form such as:
Reply: tickString tickerId=1, tickType=45, value=1446324128
Reply: tickPrice tickerId=1, field=4, price=29.15, canAutoExecute=0
Reply: tickSize tickerId=1, field=5, size=23
But when I try to isolate the price field using this modified code below, I get no response or errors of the following:
from ib.opt import ibConnection, Connection, message
import time
import ib_data_types as datatype
from ib.ext.Contract import Contract
price_field = ''
def reply_handler(msg):
print("Reply:", msg)
def my_callback_handler(msg):
if msg.field == 4:
price_field = msg.price
else:
print(msg)
def request_streaming_data(ib_conn):
# Stream market data
ib_conn.reqMktData(1,
contract,
datatype.GENERIC_TICKS_NONE,
datatype.SNAPSHOT_NONE)
time.sleep(1)
if __name__ == "__main__":
tws_conn = ibConnection(host='localhost', port=7497, clientId=100)
tws_conn.connect()
#tws_conn.registerAll(reply_handler)
tws_conn.register(my_callback_handler)
contract = Contract()
contract.m_symbol = 'GE'
contract.m_exchange = 'SMART'
contract.m_currency = 'USD'
contract.m_secType = 'STK'
request_streaming_data(tws_conn)
print(price_field)
Output to screen if tws_conn.register(my_callback_handler):
Server Version: 76
TWS Time at connection:20151031 13:50:06 PST
Process finished with exit code 0
Output to screen if tws_conn.registerAll(my_callback_handler):
31-Oct-15 13:56:33 ERROR Exception in message dispatch.
Handler 'my_callback_handler' for 'tickString'
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\ib\opt\dispatcher.py", line 44, in call
results.append(listener(message))
File "C:/Users/Admin/PycharmProjects/TestStrat/4.py", line 11, in my_callback_handler
if msg.field == 4:
AttributeError: 'TickString' object has no attribute 'field'
Lastly I have tested the code from this question here:
IbPy: How to extract API response into a variable
Output:
Server Version: 76
TWS Time at connection:20151031 13:53:51 PST
Price - field 4: 29.31
Process finished with exit code 0
And I get the price field correctly. But as far I can tell, my implementations are similar. Why am I not getting the response? Thanks!

You're trying to parse all messages in your reply handler and making the wrong assumption that they all have a field called field. If you look at your earlier response, TickString doesn't have field.
You need to either send only TickPrice messages or check in your handler if it's really a TickPrice message.
In the example you linked to, notice the code self.tws.register(self.tickPriceHandler, 'TickPrice'). That's making a special handler for just the TickPrice messages.