Develop Reference

To find Oplog size using python

How to find oplog size in mongodb using python?
For example :
replGetSetStatus is equivalent to rs.status()
Is there any similar command to find rs.printReplicationInfo()
uri = "mongodb://usernamen:password#host:port/admin"
conn = pymongo.MongoClient(uri)
db = conn['admin']
db_stats = db.command({'replSetGetStatus' :1})
primary_optime = 0
secondary_optime = 0
for key in db_stats['members'] :
if key['stateStr'] == 'SECONDARY' :
secondary_optime = key['optimeDate']
if key['stateStr'] == 'PRIMARY' :
primary_optime =key['optimeDate']
print 'primary_optime : ' + str(primary_optime)
print 'secondary_optime : ' + str(secondary_optime)
seconds_lag = (primary_optime - secondary_optime ).total_seconds()
#total_seconds() userd to get the lag in seconds rather than datetime object
print 'secondary_lag : ' + str(seconds_lag)
This is my code. The db.command({'replSetGetStatus' :1}) is working.
Similarly I need for the oplog size.

The following commands executed from any replicaSet member will give you the size of oplog:
Uncompressed size in MB:
db.getReplicationInfo().logSizeMB
Uncompressed current size in Bytes:
db.getSiblingDB('local').oplog.rs.stats().size
Compressed current size in Bytes:
db.getSiblingDB('local').oplog.rs.stats().storageSize
Max configured size:
db.getSiblingDB('local').oplog.rs.stats().maxSize

How to parse Log file to object list

I'm working with data tipe Log (ROS).
Multiple objects are saved in Log file like this:
header:
seq: 2
stamp:
secs: 1596526199
nsecs: 140017032
frame_id: ''
level: 2
name: "/replicator_node"
msg: "Replicator node dumping to /tmp/replicator_dumps"
file: "replicator_node.py"
function: "__init__"
line: 218
topics: [/move_mongodb_entries/status, /move_mongodb_entries/goal, /move_mongodb_entries/result,
/move_mongodb_entries/cancel, /rosout, /move_mongodb_entries/feedback]
header:
seq: 2
stamp:
secs: 1596526198
nsecs: 848793029
frame_id: ''
level: 2
name: "/mongo_server"
msg: "2020-08-04T09:29:58.848+0200 [initandlisten] connection accepted from 127.0.0.1:58672\
\ #1 (1 connection now open)"
file: "mongodb_server.py"
function: "_mongo_loop"
line: 139
topics: [/rosout]
As you can see not everything is in same line as it's name.
I want to pars it to have object list - so I could access it like that:
object[1].msg would give me:
"2020-08-04T09:29:58.848+0200 [initandlisten] connection accepted from 127.0.0.1:58672 #1 (1 connection now open)"
Also, sometimes file name is something like: \home\nfoo\foo.py which results in log file as:
file: "\home
foo\foo.py"

It's an interesting exercise... Assuming that the structure is really consistent for all log entries, you can try something like this - pretty convoluted, but it works for the example in the question:
ros = """[your log above]"""
def manage_lists_2(log_ind, list_1, list_2, mystr):
if log_ind == 0:
list_1.append(mystr.split(':')[0].strip())
list_2[-log_ind].append(mystr.split(':')[1].strip())
m_keys2 = []
m_key_vals2 = [[],[]]
header_keys2 = []
header_key_vals2 = [[],[]]
stamp_keys2 = []
stamp_key_vals2 = [[],[]]
for log in logs:
for l in log.splitlines():
if l[0]!=" ":
items = [m_keys2, m_key_vals2]
elif l[0:3] != " ":
items = [header_keys2, header_key_vals2]
else:
items = [stamp_keys2, stamp_key_vals2]
manage_lists_2(logs.index(log), items[0], items[1], l)
for val in m_key_vals2:
for a, b, in zip(m_keys2,val):
print(a, ": ",b)
if a == "header":
for header_key in header_keys2:
print('\t',header_key,':',header_key_vals2[m_keys2.index(a)][header_keys2.index(header_key)])
if header_key == "stamp":
for stamp_key in stamp_keys2:
print('\t\t',stamp_key,':',stamp_key_vals2[m_keys2.index(a)][stamp_keys2.index(stamp_key)])
print('---')
Output:
header :
seq : 2
stamp :
secs : 1596526199
nsecs : 140017032
frame_id : 'one id'
level : 2
name : "/replicator_node"
msg : "Replicator node dumping to /tmp/replicator_dumps"
file : "replicator_node.py"
function : "__init__"
line : 218
topics : [/move_mongodb_entries/status, /move_mongodb_entries/goal, /move_mongodb_entries/result, /move_mongodb_entries/cancel, /rosout, /move_mongodb_entries/feedback]
---
header :
seq : 2
stamp :
secs : 1596526199
nsecs : 140017032
frame_id : 'one id'
level : 3
name : "/mongo_server"
msg : "2020-08-04T09
file : "mongodb_server.py"
function : "_mongo_loop"
line : 139
topics : [/rosout]
Having gone through that, I would recommend that - if you are going to do this on a regular basis - you find a way to store the data in xml format; it's a natural fit for it.

python myhdl package how to generate verilog initial block

From the code mostly from the sample of myhdl:
from myhdl import Signal, intbv, delay, always, now, Simulation, toVerilog
__debug = True
def ClkDriver(clk):
halfPeriod = delay(10)
#always(halfPeriod)
def driveClk():
clk.next = not clk
return driveClk
def HelloWorld(clk, outs):
counts = intbv(3)[32:]
#always(clk.posedge)
def sayHello():
outs.next = not outs
if counts >= 3 - 1:
counts.next = 0
else:
counts.next = counts + 1
if __debug__:
print "%s Hello World! outs %s %s" % (
now(), str(outs), str(outs.next))
return sayHello
clk = Signal(bool(0))
outs = Signal(intbv(0)[1:])
clkdriver_inst = ClkDriver(clk)
hello_inst = toVerilog(HelloWorld, clk, outs)
sim = Simulation(clkdriver_inst, hello_inst)
sim.run(150)
I expect it to generate a verilog program that contains an initial block, like something:
module HelloWorld(...)
reg [31:0] counts;
initial begin
counts = 32'h3
end
always #(...
How can you get the initial block generated?
Note that on the google cache for old.myhdl.org/doku.php/dev:initial_values it links to example https://bitbucket.org/cfelton/examples/src/tip/ramrom/ . So it looks the feature should be supported. However the rom sample generates static case statements. That's not what I'm looking for.

Three steps to resolve it:
Update to the latest myhdl on master or a version that contains the hash 87784ad which added the feature under issue #105 or #150. As an example for virtualenv, run a git clone, followed by pip install -e <path-to-myhdl-dir>.
Change the signal to a list.
Set toVerilog.initial_values=True before calling toVerilog.
Code snippet follows.
def HelloWorld(clk, outs):
counts = [Signal(intbv(3)[32:])]
#always(clk.posedge)
def sayHello():
outs.next = not outs
if counts[0] >= 3 - 1:
counts[0].next = 0
else:
counts[0].next = counts[0] + 1
if __debug__:
print "%s Hello World! outs %s %s %d" % (
now(), str(outs), str(outs.next), counts[0])
return sayHello
clk = Signal(bool(0))
outs = Signal(intbv(0)[1:])
clkdriver_inst = ClkDriver(clk)
toVerilog.initial_values=True
hello_inst = toVerilog(HelloWorld, clk, outs)
sim = Simulation(clkdriver_inst, hello_inst)
sim.run(150)

libvirt: From state integer to string?

I have this simple script to check the memory usage of virtual machines managed by libvirt.
How can I convert the integer for state from dom.info() to a human readable string?
import libvirt
import re
import sys
def mem_total_kb():
meminfo = open('/proc/meminfo').read()
matched = re.search(r'^MemTotal:\s+(\d+)', meminfo)
return int(matched.groups()[0])
def main():
conn = libvirt.openReadOnly(None)
if conn == None:
print 'Failed to open connection to the hypervisor'
sys.exit(1)
used_mem_sum = 0
for domain_id in conn.listDomainsID():
dom = conn.lookupByID(domain_id)
state, max_mem, used_mem, vcpus, cpu_time_used = dom.info()
print(
'name=%s state=%s max_mem=%s used_mem=%s vcpus=%s cpu_time_used=%s' % (dom.name(), state, max_mem, used_mem, vcpus, cpu_time_used))
used_mem_sum += used_mem
print('Sum of used mem: %s KiB' % used_mem_sum)
mem_total = mem_total_kb()
print('Sum of physical mem: %s KiB' % mem_total)
if used_mem_sum > mem_total:
print('########## VMs use more RAM than available!')
return
mem_left=mem_total - used_mem_sum
print('Memory left: %s KiB' % (mem_left))
mem_left_should=4000000
if mem_left<mem_left_should:
print('less than mem_left_should=%sKiB left!' % mem_left_should)
if __name__ == '__main__':
main()
Docs: https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainInfo
state: the running state, one of virDomainState
enum virDomainState {
VIR_DOMAIN_NOSTATE = 0
no state
VIR_DOMAIN_RUNNING = 1
the domain is running
VIR_DOMAIN_BLOCKED = 2
the domain is blocked on resource
VIR_DOMAIN_PAUSED = 3
the domain is paused by user
VIR_DOMAIN_SHUTDOWN = 4
the domain is being shut down
VIR_DOMAIN_SHUTOFF = 5
the domain is shut off
VIR_DOMAIN_CRASHED = 6
the domain is crashed
VIR_DOMAIN_PMSUSPENDED = 7
the domain is suspended by guest power management
VIR_DOMAIN_LAST = 8
NB: this enum value will increase over time as new events are added to the libvirt API. It reflects the last state supported by this version of the libvirt API.
}

Looks like at least the enum constant names are exposed in the libvirt module. The following works for me...
import libvirt
import pprint
import re
d = {}
for k, v in libvirt.__dict__.iteritems():
if re.match('VIR_DOMAIN_[A-Z]+$', k):
d[v] = k
pprint.pprint(d)
...and prints...
{0: 'VIR_DOMAIN_NOSTATE',
1: 'VIR_DOMAIN_RUNNING',
2: 'VIR_DOMAIN_BLOCKED',
3: 'VIR_DOMAIN_PAUSED',
4: 'VIR_DOMAIN_SHUTDOWN',
5: 'VIR_DOMAIN_SHUTOFF',
6: 'VIR_DOMAIN_CRASHED',
7: 'VIR_DOMAIN_PMSUSPENDED'}
The descriptions, which are likely just comments in the original source code, are probably not exposed. One of the examples defines its own dictionary on line 106...
state_names = { libvirt.VIR_DOMAIN_RUNNING : "running",
libvirt.VIR_DOMAIN_BLOCKED : "idle",
libvirt.VIR_DOMAIN_PAUSED : "paused",
libvirt.VIR_DOMAIN_SHUTDOWN : "in shutdown",
libvirt.VIR_DOMAIN_SHUTOFF : "shut off",
libvirt.VIR_DOMAIN_CRASHED : "crashed",
libvirt.VIR_DOMAIN_NOSTATE : "no state" }
...so you could just take it from there, although it seems to be incomplete.

This bash script will fetch the list of virDomainState values (excluding VIR_DOMAIN_LAST) and their descriptions in JSON format:
#!/bin/bash
HEADER=include/libvirt/libvirt-domain.h
get_header_file()
{
curl -s https://raw.githubusercontent.com/libvirt/libvirt/master/"$HEADER"
}
select_virDomainState_block()
{
sed -n '/virDomainState:/,/^\} virDomainState;/ p'
}
join_multiline_comments()
{
sed -n '\%/\*[^*]*$% N; \%/\*.*\*/$% { s/\s*\n\s*/ /; p; }'
}
enum_values_to_json_map()
{
echo '{'
sed "s/\s*VIR_DOMAIN\S\+\s*=\s*//; s^,\s*/\*\s*^ : '^; s^\s*\*/^',^;"
echo '}'
}
get_header_file \
| select_virDomainState_block \
| join_multiline_comments \
| grep 'VIR_DOMAIN\S\+\s*=' \
| enum_values_to_json_map
Example usage:
$ ./get_libvirt_domain_states
{
0 : 'no state',
1 : 'the domain is running',
2 : 'the domain is blocked on resource',
3 : 'the domain is paused by user',
4 : 'the domain is being shut down',
5 : 'the domain is shut off',
6 : 'the domain is crashed',
7 : 'the domain is suspended by guest power management',
}
Note that the script downloads a ~150KB file from the libvirt mirror repository at GitHub. It is intended for facilitating staying up-to-date with the libvirt code. Of course you can call it from your python code, but personally I wouldn't do that.

python return list from linux command output

I am new to python and I'm learning rapidly, but this is beyond my current level of understanding. I'm trying to to pull the output from the linux command apcaccess into a list in python.
apcaccess is a linux command to get the status of an APC UPS. The output is this:
$ apcaccess
APC : 001,035,0933
DATE : 2014-11-12 13:38:27 -0500
HOSTNAME : doormon
VERSION : 3.14.10 (13 September 2011) debian
UPSNAME : UPS
CABLE : USB Cable
DRIVER : USB UPS Driver
UPSMODE : Stand Alone
STARTTIME: 2014-11-12 12:28:00 -0500
MODEL : Back-UPS ES 550G
STATUS : ONLINE
LINEV : 118.0 Volts
LOADPCT : 15.0 Percent Load Capacity
BCHARGE : 100.0 Percent
TIMELEFT : 46.0 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME : 0 Seconds
SENSE : Medium
LOTRANS : 092.0 Volts
HITRANS : 139.0 Volts
ALARMDEL : 30 seconds
BATTV : 13.6 Volts
LASTXFER : No transfers since turnon
NUMXFERS : 2
XONBATT : 2014-11-12 12:33:35 -0500
TONBATT : 0 seconds
CUMONBATT: 53 seconds
XOFFBATT : 2014-11-12 12:33:43 -0500
STATFLAG : 0x07000008 Status Flag
SERIALNO : 4B1335P17084
BATTDATE : 2013-08-28
NOMINV : 120 Volts
NOMBATTV : 12.0 Volts
FIRMWARE : 904.W1 .D USB FW:W1
END APC : 2014-11-12 13:38:53 -0500
I've tried different iterations of Popen such as:
def check_apc_ups():
output = subprocess.Popen("apcaccess", stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
x1, x2, x3, x4, x5 = output
I would like to be able to pull each line into a list or tuple containing all 32 and then only display/print what I need, such as TIMELEFT and BCHARGE.
Any help would be greatly appreciated.

There are already answers how to get the output of the command into python.
It is not clear what you are going to do with the output. Maybe, a dictionary (dict) is better than a list for you:
# stolen from Hackaholic's answer
import subprocess
child = subprocess.Popen('apcaccess',stdout=subprocess.PIPE)
msg,err = child.communicate()
# now create the dict:
myDict={}
#for i in msg.split("\n"): # loop over lines
for i in msg.splitlines(): # EDIT: See comments
splitted=i.split(":") # list like ["HOSTNAME ", " doormon"]
# remove leading & trailing spaces, add to dict
myDict[splitted[0].strip()]=splitted[1].strip()
#Now, you can easily access the items:
print myDict["SERIALNO"]
print myDict["STATUS"]
print myDict["BATTV"]
for k in myDict.keys():
print k +" = "+ myDict[k]

from subprocess import check_output
out = check_output(["apcaccess"])
spl = [ele.split(":",1)for ele in out.splitlines()]
d = {k.rstrip():v.lstrip() for k,v in spl}
print(d['BCHARGE'])
print(d["TIMELEFT"])
100.0 Percent
46.0 Minutes
from subprocess import check_output
def get_apa():
out = check_output(["apcaccess"])
spl = [ele.split(":", 1) for ele in out.splitlines()]
d = {k.rstrip(): v.lstrip() for k, v in spl}
return d
output = get_apa()
print (output['BCHARGE'])
100.0 Percent
To print all key/values pairings:
for k,v in get_apa().items():
print("{} = {}".format(k,v))

what you need is subprocess module
import subprocess
child = subprocess.Popen('apcaccess',stdout=subprocess.PIPE)
msg,err = child.communicate()
print(msg.split())