Develop Reference

Retrieving FMU co-simulation results when using Master in PyFMI

I am trying to co-simulate two Matlab-Simulink FMUs using the Master from PyFMI:
from pyfmi import Master
""" Loading FMUs, establishing connection between models, defining Master simulator and simulation options """
dummy_model = load_fmu('dummy.fmu', log_level = 4)
batt_model = load_fmu('battery.fmu', log_level = 4)
models = [dummy_model, batt_model]
connection = [(dummy_model, 'P_to_batt_fmu', batt_model, 'P_to_batt_fmu')]
mastersim = Master(models, connection)
opts = mastersim.simulate_options()
opts['step_size'] = 1000
opts['result_handling'] = 'memory'
""" Simulating """
res = mastersim.simulate(final_time = 1000, options = opts)
Everything until here runs as expected. The problem comes when trying to retrieve the results from res, since I cannot call the names of the output variables in my Simulink models. These have names like 'Ibatt', 'SoC' or 'Qbatt' (see image below), and I would expect them to be the keys of res.
Output variable names in Simulink
But they aren't. In fact, res appears to be a python dictionary with very strange keys and values, and until now I have been unable to make heads or tails of it:
{0: <pyfmi.common.algorithm_drivers.AssimuloSimResult object at 0x000002B3AC3CCF10>,
<pyfmi.fmi.FMUModelCS2 object at 0x000002B3AB8BC800>: <pyfmi.common.algorithm_drivers.AssimuloSimResult object at 0x000002B3AC3CCF10>,
1: <pyfmi.common.algorithm_drivers.AssimuloSimResult object at 0x000002B3AC3CE1A0>,
<pyfmi.fmi.FMUModelCS2 object at 0x000002B3AB889670>: <pyfmi.common.algorithm_drivers.AssimuloSimResult object at 0x000002B3AC3CE1A0>}
As I said before, I would expect res to be a dictionary with very straightforward keys, which is what happens when simulating just one of the FMUs:
batt_model = load_fmu('battery.fmu', log_level = 4)
opts = batt_model.simulate_options()
opts['ncp'] = 1000
res = batt_model.simulate(final_time=1000, options = opts)
If I run the code above, then res.keys() are just the names of my Simulink output variables.
My questions are:
How can I find my variables in res when using Master?
How can I work with them in an easy way?
So far I have been able to work with a results.txt file by stating opts['result_handling'] = 'file', however I would much prefer storing and working with the results in 'memory'

C.GoString() only returning first character

I'm trying to call a Go function from Python using c-shared (.so) file. In my python code I'm calling the function like this:
website = "https://draftss.com"
domain = "draftss.com"
website_ip = "23.xxx.xxx.xxx"
website_tech_finder_lib = cdll.LoadLibrary("website_tech_finder/builds/websiteTechFinder.so")
result_json_string: str = website_tech_finder_lib.FetchAllData(website, domain, website_ip)
On Go side I'm converting the strings to Go strings based on this SO post (out of memory panic while accessing a function from a shared library):
func FetchAllData(w *C.char, d *C.char, dIP *C.char) *C.char {
var website string = C.GoString(w)
var domain string = C.GoString(d)
var domainIP string = C.GoString(dIP)
fmt.Println(website)
fmt.Println(domain)
fmt.Println(domainIP)
.... // Rest of the code
}
The website domain and domainIP just have the first characters of the strings that I passed:
fmt.Println(website) // -> h
fmt.Println(domain) // -> d
fmt.Println(domainIP) // -> 2
I'm a bit new to Go, so I'm not sure if I'm doing something stupid here. How do I get the full string that I passed?

You need to convert the parameters as UTF8 bytes.
website = "https://draftss.com".encode('utf-8')
domain = "draftss.com".encode('utf-8')
website_ip = "23.xxx.xxx.xxx".encode('utf-8')
lib = cdll.LoadLibrary("website_tech_finder/builds/websiteTechFinder.so")
result_json_string: str = website_tech_finder_lib.FetchAllData(website, domain, website_ip)

python interaction with BPF maps

I'm wondering if there is an easy to to initialize BPF maps from python userspace. For my project, I'll have a scary looking NxN 2d array of floats for each process. For simplicity's sake, lets assume N is constant across processes (say 5). To achieve kernel support for this data, I could do something like:
b = BPF(text = """
typedef struct
{
float transMat[5][5];
} trans_struct;
BPF_HASH(trans_mapping, char[16], trans_struct);
.....
""")
I'm wondering if theres an easy way to initialize this map from python. Something like:
for ele in someDictionary:
#asume someDitionary has mapping (comm -> 5x5 float matrix)
b["trans_mapping"].insert(ele, someDictionary[ele])
I suppose at the crux of my confusion is -- 1) are all map methods available to the user, 2) how do we ensure type consistenty when going from python objects to c structures

Solution based on pchaigno's comment -- The key things to note are the use of c_types to ensure type consistency across environments, and extracting the table by indexing the BPF program object. Due to our ability to get maps by indexing, the get_table() function is now considered out of date. This format provides the structure of loading data into the map from the python front-end, but doesn't completely conform with the specifics of my question.
from time import sleep, strftime
from bcc import BPF
from bcc.utils import printb
from bcc.syscall import syscall_name, syscalls
from ctypes import *
b = BPF(text = """
BPF_HASH(start, u32, u64);
TRACEPOINT_PROBE(raw_syscalls, sys_exit)
{
u32 syscall_id = args->id;
u32 key = 1;
u64 *val;
u32 uid = bpf_get_current_uid_gid();
if (uid == 0)
{
val = start.lookup(&key); //find value associated with key 1
if (val)
bpf_trace_printk("Hello world, I have value %d!\\n", *val);
}
return 0;
}
""")
thisStart = b["start"]
thisStart[c_int(1)] = c_int(9) #insert key-value part 1->9
while 1:
try:
(task, pid, cpu, flags, ts, msg) = b.trace_fields()
except KeyboardInterrupt:
print("Detaching")
exit()
print("%-18.9f %-16s %-6d %s" % (ts, task, pid, msg))

How to get the latest offset value from a confluent_python AVRO consumer

I am pretty new to confluent_kafka but I've gained some experience with kafka-python. What I am trying to do is changing the offset where to start consuming messages. This why I'd like to build a consumer client able to move back to previous messages in order to return data that will populate a dashboard. Said that using the kafka-python package I can use the seek_to_end (https://github.com/dpkp/kafka-python/blob/c0fddbd24269d4333e3b6630a23e86ffe33dfcb6/kafka/consumer/group.py#L788) method in order to get the position value of the latest commit. Having that I can subtract values and get back to the previous messages using the seek method (https://github.com/dpkp/kafka-python/blob/c0fddbd24269d4333e3b6630a23e86ffe33dfcb6/kafka/consumer/group.py#L738)
On the other hand, the conflient_kafka seems to have no similar functions and what I've found so far is to use the variables OFFSET_END which has a value of -1 and it doesn't return me the offset numeric value of the latest and largest one. I can use the 'seek' function as well, but I need a way to have the numeric value of the latest offset and not -1.
My avro consumer looks like
from confluent_kafka.avro import AvroConsumer
if __name__ == '__main__':
c = AvroConsumer({"bootstrap.servers": "locahost:29092", "group.id":"mygroup",'schema.registry.url': 'http://localhost:8081',
'enable.auto.commit': True,'default.topic.config': {'auto.offset.reset': 'smallest'}})
def my_assign (consumer, partitions):
for p in partitions:
p.offset = confluent_kafka.OFFSET_END
print("offset=",p.offset)
print('assign', partitions)
print('position:',consumer.position(partitions))
consumer.assign(partitions)
c.subscribe(["mytopic"],on_assign=my_assign)
while True:
m = c.poll(1)
if m is None:
continue
if m.error() is None:
print('Received message', m.value(),m.offset())
c.close()
which produces the following result:
offset= -1
assign [TopicPartition{topic=mytopic,partition=0,offset=-1,error=None}]
position: [TopicPartition{topic=mytopic,partition=0,offset=-1001,error=None}]
and stays waiting for the next message. I was wondering if someone can help me out. Thanks

You can use Consumer.get_watermark_offsets (see docs)
Example:
cfg = {
# ... ...
"group.id": str(uuid4())
}
consumer = AvroConsumer(cfg)
topic_partition = TopicPartition("topic-name", partition=123)
low, high = consumer.get_watermark_offsets(topic_partition)
print("the latest offset is {}".format(high))

Dynamically create objects and setting their attributes using suds lib in python

I'm using a raspberry to send some sensor data over to a SOAP webservice. RPi gets the data from the serial port. The format is 'DATE, VALUE1, VALUE2, VALUE3, VALUE4\r\n'. The webservice request looks like this
<sensors>
<Sensor>
<SensorId>int</SensorId>
<NodeId>int</NodeId>
<SensorTypeId>int</SensorTypeId>
<Value>double</Value>
<Status>string</Status>
<Date>dateTime</Date>
<Deleted>boolean</Deleted>
<Updated>boolean</Updated>
<RemoteId>int</RemoteId>
<DateOfLastUpdate>dateTime</DateOfLastUpdate>
<UserId>int</UserId>
<ErrorMessage>string</ErrorMessage>
</Sensor>
<Sensor>
<SensorId>int</SensorId>
<NodeId>int</NodeId>
<SensorTypeId>int</SensorTypeId>
<Value>double</Value>
<Status>string</Status>
<Date>dateTime</Date>
<Deleted>boolean</Deleted>
<Updated>boolean</Updated>
<RemoteId>int</RemoteId>
<DateOfLastUpdate>dateTime</DateOfLastUpdate>
<UserId>int</UserId>
<ErrorMessage>string</ErrorMessage>
</Sensor>
</sensors>
<username>string</username>
<password>string</password>
<UniqueClientID>string</UniqueClientID>
<project>string</project>
Each line I get from the serial port has 4 sensor values and the datetime these values were logged. So I need to create 4 objects with the SUDS library method client.factory.create() for each line I parse from the serial, add the values to each attribute and append() the objects to the Array of Sensor objects that the webservice accepts as its first parameter. The problem is I can't find a way to dynamically create the objects, enter the attributes' values, and append them to the big array. I'm going to parse probably 600lines from the serial port, so 4x600=2400 will need to be created. Hard-coding object names like this
while True:
serial_str = port.readline()
if serial_str:
string_list = serial_str.split(',')
date = 'T'.join( [ string_list[i] for i in [0, 1] ] )
temperature = string_list[2]
humidity = string_list[3]
rain = string_list[4]
wind = string_list[5].replace("\r\n","")
Sensor_obj1 = client.factory.create('Sensor')
Sensor_obj1.SensorId = -1
Sensor_obj1.NodeId = 1
Sensor_obj1.SensorTypeId = 2
Sensor_obj1.Value = temperature
Sensor_obj1.Status = ''
Sensor_obj1.Date = date
Sensor_obj1.Deleted = 0
Sensor_obj1.Updated = 0
Sensor_obj1.RemoteId = 0
Sensor_obj1.DateOfLastUpdate = datetime.now().strftime('%Y-%m-%dT%H:%M:%S')datetime.now()
Sensor_obj1.UserId = 0
Sensor_obj1.ErrorMessage = ''
Sensor_list_obj.Sensor.append(Sensor_obj1)
Sensor_obj2 = client.factory.create('Sensor')
...
would work if I only had 1 line to send, but even then its a bad programming style. I just got started with python 2.x, so I would appreciate if someone would point me to the right direction here. Thanks in advance

I am not very familiar with the problem at hand, but I would guess that you can add more parameters to the call to client.factory.create('Sensor') call. Does it accept, **kwargs, so that you can call:
client.factory.create('Sensor', SensorID=-1, NodeId=1, ...)
If not, you can always abstract the initialization by:
Creating a function that does all the asignments and hides away the 'gory' details. This is elegant enough, and probably you shouldn't try anything more advanced if you have just started programming in python.
Create a class Sensor, that inherits from the one of suds. Unfortunately, I don't know enough about the class of objects 'client.factory.create'.
Also, I think I would create a list of sensors (or dictionary, if that makes sense in your code), so that you append all the sensors to that structure, otherwise your program will be impossible to use if you have three or five sensors instead of 4.