Develop Reference

How to use attributeselectedclassifier on pyweka?

Im translating a model done on weka to python-weka-wrapper3 and i dont know how to an evaluator and search options on attributeselectedclassifier.
This is the model on weka:
weka.classifiers.meta.AttributeSelectedClassifier -E "weka.attributeSelection.CfsSubsetEval -P 1 -E 1" -S "weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1" -W weka.classifiers.meta.MultiSearch -- -E FM -search "weka.core.setupgenerator.MathParameter -property classifier.classifier.classifier.numOfBoostingIterations -min 5.0 -max 50.0 -step 1.0 -base 10.0 -expression I" -class-label 1 -algorithm "weka.classifiers.meta.multisearch.DefaultSearch -sample-size 100.0 -initial-folds 2 -subsequent-folds 10 -initial-test-set . -subsequent-test-set . -num-slots 1" -log-file /Applications/weka-3-8-3 -S 1 -W weka.classifiers.meta.Bagging -- -P 100 -S 1 -num-slots 1 -I 100 -W weka.classifiers.meta.FilteredClassifier -- -F "weka.filters.supervised.instance.SMOTE -C 0 -K 3 -P 250.0 -S 1" -S 1 -W weka.classifiers.meta.CostSensitiveClassifier -- -cost-matrix "[0.0 1.0; 1.0 0.0]" -S 1 -W weka.classifiers.trees.ADTree -- -B 10 -E -3 -S 1
and I have this right now:
base = Classifier(classname="weka.classifiers.trees.ADTree", options=["-B", "10", "-E", "-3", "-S", "1"])
cls = SingleClassifierEnhancer(classname="weka.classifiers.meta.CostSensitiveClassifier",
options =["-cost-matrix", "[0.0 1.0; 1.0 0.0]", "-S", "1"])
cls.classifier = base
smote = Filter(classname="weka.filters.supervised.instance.SMOTE", options=["-C", "0", "-K", "3", "-P", "250.0", "-S", "1"])
fc = FilteredClassifier()
fc.filter = smote
fc.classifier = cls
bagging_cls = Classifier(classname="weka.classifiers.meta.Bagging",
options=["-P", "100", "-S", "1", "-num-slots", "1", "-I", "100"])
bagging_cls.classifier = fc
multisearch_cls = MultiSearch(
options = ["-S", "1"])
multisearch_cls.evaluation = "FM"
multisearch_cls.log_file = "/home/pablo/Escritorio/TFG/OUTPUT.txt"
multisearch_cls.search = ["-sample-size", "100", "-initial-folds", "2", "-subsequent-folds", "10",
"-initial-test-set", ".", "-subsequent-test-set", ".", "-num-slots", "1"]
mparam = MathParameter()
mparam.prop = "numOfBoostingIterations"
mparam.minimum = 5.0
mparam.maximum = 50.0
mparam.step = 1.0
mparam.base = 10.0
mparam.expression = "I"
multisearch_cls.parameters = [mparam]
multisearch_cls.classifier = bagging_cls
AttS_cls = AttributeSelectedClassifier()
AttS_cls.evaluator = "weka.attributeSelection.CfsSubsetEval -P 1 -E 1"
AttS_cls.search = "weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1"
AttS_cls.classifier = multisearch_cls
train, test = data_modelos_1_2.train_test_split(70.0, Random(1))
AttS_cls.build_classifier(train)
evl = Evaluation(train)
evl.crossvalidate_model(AttS_cls, test, 10, Random(1))
print(AttS_cls)
#graph.plot_dot_graph(AttS_cls.graph)
print("")
print("=== Setup ===")
print("Classifier: " + AttS_cls.to_commandline())
print("Dataset: ")
print(test.relationname)
print("")
print(evl.summary("=== " + str(10) + " -fold Cross-Validation ==="))
print(evl.class_details())
plcls.plot_roc(evl, class_index=[0, 1], wait=True)
but when I do
AttS_cls.evaluator = "weka.attributeSelection.CfsSubsetEval -P 1 -E 1"
AttS_cls.search = "weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1"
it reach me this error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/tmp/ipykernel_40724/2750622902.py in <module>
30
31 AttS_cls = AttributeSelectedClassifier()
---> 32 AttS_cls.search = "weka.attributeSelection.GreedyStepwise"
33 AttS_cls.classifier = multisearch_cls
34
/usr/local/lib/python3.8/dist-packages/weka/classifiers.py in search(self, search)
435 :type search: ASSearch
436 """
--> 437 javabridge.call(self.jobject, "setSearch", "(Lweka/attributeSelection/ASSearch;)V", search.jobject)
438
439
AttributeError: 'str' object has no attribute 'jobject'
I understand that I have to set them as objects because it raise this error because i try to set them as strings but I dont know how.

You need to instantiate ASSearch and ASEvaluation objects. If you have command-lines, you can use the from_commandline helper method like this:
from weka.core.classes import from_commandline, get_classname
from weka.attribute_selection import ASSearch
from weka.attribute_selection import ASEvaluation
search = from_commandline('weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1', classname=get_classname(ASSearch))
evaluation = from_commandline('weka.attributeSelection.CfsSubsetEval -P 1 -E 1', classname=get_classname(ASEvaluation))
The second argument of the from_commandline method is the classname of the wrapper that you want to use instead of OptionHandler. For simplicity, I import the correct wrappers and then use the get_classname method to return the dot notation of the wrapper's class. That way I can avoid accidental typos in the classname strings.
Also, by using single quotes, you won't have to worry about Weka's quotes in the command-lines and you can just use the Weka command-line string verbatim.
You can also use the same approach for instantiating the AttributeSelectedClassifier wrapper itself, without having to go through instantiating search and evaluation separately:
from weka.core.classes import from_commandline, get_classname
from weka.classifiers import AttributeSelectedClassifier
cls = from_commandline('weka.classifiers.meta.AttributeSelectedClassifier -E "weka.attributeSelection.CfsSubsetEval -P 1 -E 1" -S "weka.attributeSelection.GreedyStepwise -B -T -1.7976931348623157E308 -N -1 -num-slots 1" -W weka.classifiers.meta.MultiSearch -- -E FM -search "weka.core.setupgenerator.MathParameter -property classifier.classifier.classifier.numOfBoostingIterations -min 5.0 -max 50.0 -step 1.0 -base 10.0 -expression I" -class-label 1 -algorithm "weka.classifiers.meta.multisearch.DefaultSearch -sample-size 100.0 -initial-folds 2 -subsequent-folds 10 -initial-test-set . -subsequent-test-set . -num-slots 1" -log-file /Applications/weka-3-8-3 -S 1 -W weka.classifiers.meta.Bagging -- -P 100 -S 1 -num-slots 1 -I 100 -W weka.classifiers.meta.FilteredClassifier -- -F "weka.filters.supervised.instance.SMOTE -C 0 -K 3 -P 250.0 -S 1" -S 1 -W weka.classifiers.meta.CostSensitiveClassifier -- -cost-matrix "[0.0 1.0; 1.0 0.0]" -S 1 -W weka.classifiers.trees.ADTree -- -B 10 -E -3 -S 1', get_classname(AttributeSelectedClassifier))

Python swap opening/closing characters (parens, brackets, etc.)

Is there simple function to swapping opening/closing (directional) characters, such as (, ], <, etc.? For example:
>>> swap_direction("abcd{1>2(3]")
"abcd}1<2)3["
Alternatively, perhaps there is a method like str.upper() and str.lower() which converts all directional characters to either their opening or closing variant. This might look like:
>>> to_opening("abcd{1>2(3]")
"abcd}1<2)3["
>>> to_closing("abcd{1>2(3]")
"abcd}1>2(3]"
Is there an easier way besides chaining a bunch of str.replace() calls together?

Tl;dr
Based on replies by #jonrsharpe and #juanpa.arrivillaga, I came up with the following functions to be the best answer:
open_chars = "([{<"
close_chars = ")]}>"
swap_direction_table = str.maketrans(open_chars+close_chars, open_chars+close_chars)
def to_opening(text):
for ch_close, ch_open in zip(close_chars, open_chars):
if ch_close in text:
text = text.replace(ch_close, ch_open)
return text
def to_closing(text):
for ch_open, ch_close in zip(open_chars, close_chars):
if ch_open in text:
text = text.replace(ch_open, ch_close)
return text
def swap_direction(text):
return s.translate(swap_direction_table)
Additional Testing
For the sake of thoroughness, I did a few additional tests. Below is the complete list of functions I tried, organized by technique.
# sandbox.py
open_chars = "([{<"
close_chars = ")]}>"
to_open_table = str.maketrans(close_chars, open_chars)
to_close_table = str.maketrans(open_chars, close_chars)
swap_direction_table = str.maketrans(open_chars+close_chars, close_chars+open_chars)
def to_opening_tr(text):
return text.translate(to_open_table)
def to_closing_tr(text):
return text.translate(to_close_table)
def swap_direction_tr(text):
return text.translate(swap_direction_table)
def to_opening_tb(text):
return text.translate(str.maketrans(close_chars, open_chars))
def to_closing_tb(text):
return text.translate(str.maketrans(open_chars, close_chars))
def swap_direction_tb(text):
return text.translate(str.maketrans(open_chars+close_chars, close_chars+open_chars))
def to_opening_r(text):
return text.replace(")", "(").replace("]", "[").replace("}", "{").replace(">", "<")
def to_closing_r(text):
return text.replace("(", ")").replace("[", "]").replace("{", "}").replace("<", ">")
def to_opening_fr(text):
for ch_close, ch_open in zip(close_chars, open_chars):
if ch_close in text:
text = text.replace(ch_close, ch_open)
return text
def to_closing_fr(text):
for ch_open, ch_close in zip(open_chars, close_chars):
if ch_open in text:
text = text.replace(ch_open, ch_close)
return text
def swap_direction_fr(text):
for ch_open, ch_close in zip(open_chars, close_chars):
if ch_open in text and ch_close in text:
text = text.replace(ch_open, '$temp$').replace(ch_close, ch_open).replace('$temp$', ch_close)
elif ch_open in text:
text = text.replace(ch_open, ch_close)
elif ch_close in text:
text = text.replace(ch_close, ch_open)
return text
Here are the results from the timeit library:
to_opening_tr: 500000 loops, best of 5: 622 nsec per loop
to_opening_tb: 500000 loops, best of 5: 784 nsec per loop
to_opening_r: 1000000 loops, best of 5: 293 nsec per loop
to_opening_fr: 500000 loops, best of 5: 600 nsec per loop
to_closing_tr: 500000 loops, best of 5: 622 nsec per loop
to_closing_tb: 500000 loops, best of 5: 800 nsec per loop
to_closing_r: 1000000 loops, best of 5: 328 nsec per loop
to_closing_fr: 500000 loops, best of 5: 582 nsec per loop
swap_direction_tr: 500000 loops, best of 5: 569 nsec per loop
swap_direction_tb: 200000 loops, best of 5: 998 nsec per loop
swap_direction_fr: 500000 loops, best of 5: 937 nsec per loop
I used the following PowerShell script to automate the process a bit:
$test_str = "'abcd{1>2(3]'"
# $test_str = "'abcd{1>2(3]abcd}1>)))3[[[[abasdh[[[[[[[['" # Alternate string to try
Write-Host "to_opening_tr: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.to_opening_tr($test_str)"
Write-Host "to_opening_tb: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.to_opening_tb($test_str)"
Write-Host "to_opening_r: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.to_opening_r($test_str)"
Write-Host "to_opening_fr: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.to_opening_fr($test_str)"
Write-Host ""
Write-Host "to_closing_tr: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.to_closing_tr($test_str)"
Write-Host "to_closing_tb: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.to_closing_tb($test_str)"
Write-Host "to_closing_r: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.to_closing_r($test_str)"
Write-Host "to_closing_fr: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.to_closing_fr($test_str)"
Write-Host ""
Write-Host "swap_direction_tr: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.swap_direction_tr($test_str)"
Write-Host "swap_direction_tb: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.swap_direction_tb($test_str)"
Write-Host "swap_direction_fr: " -NoNewline
python -m timeit -s "import sandbox" "sandbox.swap_direction_fr($test_str)"
I don't want to hear about how my PowerShell sucks ;)
The algorithms with the best times in each category are the ones in the Tl;dr above. The results seem to be consistent among a few different sample strings I've tried.

MAP1 = {'}': '{', ')': '(', ']': '[', '>': '<'}
MAP2 = dict([(MAP1[k], k) for k in MAP1])
MAP3 = MAP1 + MAP2
def to_opening(s):
res = ''
for i in s:
res += MAP1.get(i, i)
return res
def to_closing(s):
res = ''
for i in s:
res += MAP2.get(i, i)
return res
def swap_direction(s):
res = ''
for i in s:
res += MAP3.get(i, i)
return res

Script to figure out the difference in disk space

I have a Linux command that shows the output of disk usage
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 2.8T 1.1T 1.7T 39% /data/disk2
/dev/sdc1 2.8T 1.1T 1.7T 41% /data/disk3
/dev/sdd1 2.8T 1.1T 1.7T 40% /data/disk4
I need help with continue this in a script (Python or Bash) to let me know if the disks are more than 5% different of each other. If they are over 5% imbalanced, I will write code to email the results. Any help would be appreciated.

How about this:
import re
s = """
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 2.8T 1.1T 1.7T 39% /data/disk2
/dev/sdc1 2.8T 1.1T 1.7T 41% /data/disk3
/dev/sdd1 2.8T 1.1T 1.7T 40% /data/disk4
"""
regex = re.compile(r'\d{1,2}%')
result = [int(a[:-1]) for a in regex.findall(s)]
# [39, 41, 40]
If you want to compare them at the end...
if max(result) - min(result) > 5:
print("Imbalanced!")
else:
print("Balanced!")
Of course, you can call os-level functions and get their output like this:
command_output = subprocess.check_output(['df', '-h']).decode('utf-8')

Can use awk, too:
df | awk '\
BEGIN { \
max=0; \
min=2000; \
} \
NR>1 { \
pf = $5; \
sub( /\%/, "", pf ); \
pf = pf + 0; \
if ( pf > max ) max = pf; \
if ( pf < min ) min = pf; \
} \
END { \
diff = ( max - min ); \
print diff \
}'

awk solution:
df -h | awk 'NR>1{ a[NR-1]=substr($5,1,length($5)-1) }
END{ asort(a); print ((a[length(a)]-a[1]) > 5? "Not good!" : "Good!") }'
The output (for your input):
Good!

Python daemon starts correctly with init-script but fails on startup

I have some python daemons for raspberry pi applications running on startup with init-scripts.
The init script runs fine from the console, starts and ends the background process correctly.
The script was made autostart with sudo insserv GartenwasserUI
It starts on startup, which is proof by having the LCD backlight on, but is not in the process list after logon. A manual start with sudo service GartenwasserUI start works immediately.
What could be wrong?
Here the script
#!/bin/sh
### BEGIN INIT INFO
# Provides: GartenwasserUI
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: GartenwasserUI acts as a mqtt client for LCD displaying and switching gpio through mqtt
# Description: Put a long description of the service here
### END INIT INFO
# Change the next 3 lines to suit where you install your script and what you want to call it
DIR=/usr/local/bin/Gartenwasser
DAEMON=$DIR/GartenwasserUI.py
DAEMON_NAME=GartenwasserUI
# Add any command line options for your daemon here
DAEMON_OPTS=""
# This next line determines what user the script runs as.
# Root generally not recommended but necessary if you are using the Raspberry Pi GPIO from Python.
DAEMON_USER=root
# The process ID of the script when it runs is stored here:
PIDFILE=/var/run/$DAEMON_NAME.pid
. /lib/lsb/init-functions
do_start () {
log_daemon_msg "Starting system $DAEMON_NAME daemon"
cd $DIR
#python ./$DAEMON_NAME.py &
start-stop-daemon --start --background --pidfile $PIDFILE --make-pidfile --user $DAEMON_USER --chuid $DAEMON_USER --startas $DAEMON -- $DAEMON_OPTS --verbose -stdout /var/log/GartenwasserUI.log
log_end_msg $?
}
do_stop () {
log_daemon_msg "Stopping system $DAEMON_NAME daemon"
start-stop-daemon --stop --pidfile $PIDFILE --retry 10
log_end_msg $?
}
case "$1" in
start|stop)
do_${1}
;;
restart|reload|force-reload)
do_stop
do_start
;;
status)
status_of_proc "$DAEMON_NAME" "$DAEMON" && exit 0 || exit $?
;;
*)
echo "Usage: /etc/init.d/$DAEMON_NAME {start|stop|restart|status}"
exit 1
;;
esac
exit 0
And the script itself:
#!/usr/bin/env python
__author__ = "Bernd Gewehr"
# import python libraries
import os
import signal
import sys
import time
# import libraries
import lib_mqtt as MQTT
from Adafruit_CharLCDPlate import Adafruit_CharLCDPlate
#DEBUG = False
DEBUG = True
MQTT_TOPIC_IN = "/Gartenwasser/#"
MQTT_TOPIC = "/Gartenwasser"
MQTT_QOS = 0
VALVE_STATE = [0, 0, 0, 0, 0]
def on_message(mosq, obj, msg):
"""
Handle incoming messages
"""
topicparts = msg.topic.split("/")
if DEBUG:
print msg.topic
print topicparts
for i in range(0,len(topicparts)):
print i, topicparts[i]
print msg.payload
pin = int('0' + topicparts[len(topicparts) - 1])
value = int(msg.payload)
if topicparts[2] == "in":
if pin == 29:
VALVE_STATE[0] = value
if pin == 31:
VALVE_STATE[1] = value
if pin == 33:
VALVE_STATE[2] = value
if pin == 35:
VALVE_STATE[3] = value
Message = 'V1: ' + str(VALVE_STATE[0]) + ' V2: ' + str(VALVE_STATE[1]) + '\nV3: ' + str(VALVE_STATE[2]) + ' V4: ' + str(VALVE_STATE[3])
lcd.clear()
lcd.message(Message)
# End of MQTT callbacks
def cleanup(signum, frame):
"""
Signal handler to ensure we disconnect cleanly
in the event of a SIGTERM or SIGINT.
"""
# Cleanup modules
MQTT.cleanup()
lcd.stop()
# Exit from application
sys.exit(signum)
def loop():
"""
The main loop in which we mow the lawn.
"""
while True:
time.sleep(0.08)
buttonState = lcd.buttons()
for b in btn:
if (buttonState & (1 << b[0])) != 0:
if DEBUG: print 'Button pressed for GPIO ' + str(b[1])
if b[1] > 0: MQTT.mqttc.publish(MQTT_TOPIC + '/in/' + str(b[1]), abs(VALVE_STATE[b[2]]-1), qos=0, retain=True)
time.sleep(.5)
break
# Use the signal module to handle signals
for sig in [signal.SIGTERM, signal.SIGINT, signal.SIGHUP, signal.SIGQUIT]:
signal.signal(sig, cleanup)
# Initialise our libraries
lcd = Adafruit_CharLCDPlate()
lcd.backlight(True)
MQTT.init()
MQTT.mqttc.on_message = on_message
MQTT.mqttc.subscribe(MQTT_TOPIC_IN, qos=MQTT_QOS)
# Clear display and show greeting, pause 1 sec
lcd.clear()
lcd.message("Gartenwasser\nstartet...")
time.sleep(1)
Message = 'V1: ' + str(VALVE_STATE[0]) + ' V2: ' + str(VALVE_STATE[1]) + '\nV3: ' + str(VALVE_STATE[2]) + ' V4: ' + str(VALVE_STATE[3])
lcd.clear()
lcd.message(Message)
# Cycle through backlight colors
#col = (lcd.RED, lcd.YELLOW, lcd.GREEN, lcd.TEAL,
# lcd.BLUE, lcd.VIOLET, lcd.WHITE, lcd.OFF)
#for c in col:
# lcd.ledRGB(c)
# sleep(.5)
# assign GPIO & Status index of VALVA_STATUS
btn = ((lcd.LEFT, 29, 0),
(lcd.UP, 31, 1),
(lcd.DOWN, 33, 2),
(lcd.RIGHT, 35, 3),
(lcd.SELECT, 0, 4))
# start main procedure
loop()

I found the solution:
When the machine starts up, the mosquitto service starts withs S02. Since the python daemons init scripts had no info that they depend on mosquitto, they were started as S02 also.
The change that solved the problem:
Set the dependencies in the LSB header of the init script:
# Required-Start: $remote_fs $syslog mosquitto
After that, both python daemons were executed correctly as S03.

subprocess.call where is stdout on windows?

I am using subprocess.call to run an executable from a python script
works fine under linux, the stdout from the executable is printed to the terminal as desired
when running the same subprocess.call on windows (7) the command executes successfully but i get no stdout to the cmd window
my subprocess.call looks like this:
call([self.test_dict["exe"],
"-p",
self.test_dict["pulse_input"],
"-l",
self.test_dict["library_input"],
"-c",
self.test_dict["config"],
"-r",
self.test_dict["pulse_output"],
"-s",
self.test_dict["stream_output"],
"-u",
self.test_dict["library_output"],
track_output_flag,
track_output_string,
socket_output_flag,
socket_output_string,
"-f",
self.test_dict["frame_size"],
"-d",
self.test_dict["ddi_enable"],
self.test_dict["verbose"],
])
how can i get the executables stdout displayed in the windows cmd window, i.e. how can i get the same behaviour i observe under linux?
I can now see some output using the Popen approach suggested below. my Popen call looks like this:
print(subprocess.Popen([self.test_dict["exe"],
"-p",
self.test_dict["pulse_input"],
"-l",
self.test_dict["library_input"],
"-c",
self.test_dict["config"],
"-r",
self.test_dict["pulse_output"],
"-s",
self.test_dict["stream_output"],
"-u",
self.test_dict["library_output"],
track_output_flag,
track_output_string,
socket_output_flag,
socket_output_string,
"-f",
self.test_dict["frame_size"],
"-d",
self.test_dict["ddi_enable"],
self.test_dict["verbose"]],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate())
On linux:
(' Test Harness - Version 2.2.2\n \n\nFrame 1 of 5: 1000 of 4032
pdws Rx: 0.206, G2: 32.337 (kpps) lib: ddi_n 0 residue: 11 \nFrame 2
of 5: 2000 of 4032 pdws Rx: 0.174, G2: 35.396 (kpps) lib: ddi_n 0
residue: 18 \nFrame 3 of 5: 3000 of 4032 pdws Rx: 0.197, G2: 31.913
(kpps) lib: ddi_n 0 residue: 25 \nFrame 4 of 5: 4000 of 4032 pdws Rx:
0.139, G2: 33.908 (kpps) lib: ddi_n 0 residue: 13 \nFrame 5 of 5: 4032 of 4032 pdws Rx: 2.581, G2: 20.164 (kpps) lib: ddi_n 0 residue: 0
\nlibrary status: lib.n: 52 lib.ddi_n: 0 lib.orig_n: 52\nelapsed time:
0.300253 (s)\n\n', '')
On Windows:
('', '')
the first bit of the linux output is what i would like to see printed to stdout on windows
i was mistaken that adding stdout=subprocess.PIPE was fixing the problem on windows, it wasn't - apologies for the confusion