Develop Reference

Creation of PhantomJS webdriver causes user input problems in Python over ssh session

User input to my Python program does not respond most of the time when I run it remotely and try to use a PhantomJS webdriver. Everything executes fine, same as it does when I run it locally, except the majority of keypresses don't register at a prompt from raw_input(). I have to hit a key (on average) three times for it to actually show up in the console.
This is only on raw_input in my program after it creates a PhantomJS instance. I singled out the issue by setting w = None, where w is the webdriver. After doing this, user input continued working as it should throughout the program.
I can start another ssh session and stdin works fine. I can also start another instance of my program (python run.py) and before that instance creates a webdriver, raw_input works fine in it as well. I have tried running top in another ssh session and there is no load at all on my server. So only input to the parent python process is affected.
I figure best case scenario, someone can help me get to the root of the problem. I don't have the tools to narrow down the cause of this. Thanks!
Additional info that might be relevant
ps -aux in an ssh session gives me these related processes:
[TTY: pts/1, STAT: S+] python run.py
[TTY: pts/1, STAT: S1+] node /usr/local/bin/phantomjs --webdriver=52129
[TTY: pts/1, STAT: S1+] /usr/local/lib/node_modules/phantomjs/lib/phantom/bin/phantomjs --webdriver=52129
All of them using negligible mem and cpu
Selenium webdriver creation is customized- (summarized) customdriver.py:
def mkDriver(params):
# params aren't relevant because this is the setup I always use on the server
w = webdriver.PhantomJS({'phantomjs.page.settings.loadImages': 'false'})
loginToFacebook(w)
loginToTwitter(w)
# for later- w.quit() leaves phantom process running
w.phantomjsPID = collectPhantomjsPID()
...Though I don't think the problem relates to any of the other statements there, this function finishes execution and my program continues.
Some notes:
laptop: OS X El Capitan, 2.4GHz intel core i5, 8GB memory
server: Ubuntu Server 14.04, 2.0GHz intel xeon e5, 8GB memory - I was using a micro .ec2 I set up myself, but someone set this up for me to give me something more powerful to play with. It's running on IBM Softlayer.
Before I asked: read about stdin and stdout over ssh, about paramiko as a solution (wouldn't give the behavior I wanted), checked if there is a better python function to use than raw_input, differences with psuedo-tty on and off upon ssh (don't think this applies to how I am running my program), and tried using the -u flag with python.
Initially all raw_input() statements gave an EOFError when I ran my program on the server. I "fixed" this using this solution I found, which was to import readline in all modules using raw_input. No idea why it worked. I know raw_input() is derived from sys.stdin.readline...
If you're looking at the pseudocode below, the purpose of this was to make changes to the ui and the functions it calls, then continue to test, without ever having to regenerate anything in resources (most importantly the webdriver, which takes a long time to load). The structure also makes it easy to have a few pre-written tests.
Overview of the modules...
run.py:
params = raw_input('enter parameters')
w = customdriver.mkDriver(params)
t = tweepy_connection()
f = facebook_Graph_API_conection()
s = sqlAlchemy_engine()
resources = [w, t, f, s]
while True:
choice = raw_input('1: break, 2: restart ui')
if choice == '1':
break
elif choice == '2':
reload(ui.py)
ui.ui_function(resources)
else:
continue
ui.py:
def ui_function(resources):
while True:
choice = raw_input('1: break, 2: functionA, 3: functionB')
if choice == '1':
break
elif choice == '2':
reload(file1)
params = raw_input('enter parameters')
file1.functionA(resources, params)
elif choice == '3':
reload(file2)
params = raw_input('enter some other parameters')
file2.functionB(resources, params)
else:
continue

Do the raw inputs hang when you only create a PhantomJS instance or do you do any headless browsing with it? If you're browsing, it would make sense that raw_inputs hang as each website request will consequently stall the shell.
Have you considered multi-threading it?

DEBUG only one function (or module), RUN rest of the program in Pycharm?

I will try to explain my question on simple example program (my problem is much more complex because my program is much more complex).
Lets suppose i have a program that has 2 lines, making 2 functions:
data = long_one() #takes 2 hours in DEBUG mode, 15min in RUN mode
short_one(data) #i want to DEBUG this one
Lets also say that it is very difficult to prepare the data variable and the only way to obtain it is by running the function long_one().
Is there a way to RUN long_one() and DEBUG short_one() in Pycharm?
In other words is there a way to perform either:
DEBUG with specification that long_one() should be processed in RUN mode
or RUN with specification that short_one() should be debugged?

As Asagen has proposed:
attached debugger to python console.
started my script in RUN mode
while script was running I did Tools/Attach to Process and have chosen my process.
The debugger has started from the moment i did this and stopped on first breakpoint it encountered.
There was one inconvenience - I had to know when to start debugging (in which moment to attach debugger to process). I propose a workaround:
Add to code an infininte loop in place you want to start debugging (see below):
data = long_one() # takes 2 hours in DEBUG mode, 15min in RUN mode
infinite_loop = True
print "OK man, it is the time to start debugging!"
while infinite_loop:
time.sleep(0.2) # add breakpoint here
short_one(data) #i want to DEBUG this one
Add a breakpoint inside the while loop
While running process, when you see in console the printed text "OK man, it is the time to start debugging!", attach debugger to process.
Next when it stops in the infinite loop, evaluate code fragment infinite_loop = False, so you leave the loop
It is it, you are now in DEBUG mode after running whole code before,
If you want to get back to RUN mode, just stop debugger. It is possible to switch between RUN and DEBUG as many times and in any places you want

import sys
def do_the_thing_for_debug():
print('doing the debug version')
def do_the_thing():
print('doing the release version')
if __name__ == '__main__':
"""
Detecting if you're in the PyCharm debugger or not
WARNING: This is a hacky & there is probably a better
way of doing this by looking at other items in the
global() namespace or environment for things that
begin with PyCharm and stuff.
Also, this could break in future versions of PyCharm!
If you put a breakpoint HERE and look at the callstack you will
see the entry point is in 'pydevd.py'
In debug mode, it copies off sys.argv: "sys.original_argv = sys.argv[:]"
We abuse this knowledge to test for the PyCharm debugger.
"""
if hasattr(sys, 'original_argv'):
do_the_thing = do_the_thing_for_debug
do_the_thing()
Output from PyCharm when I "run" it
doing the release version
Output from PyCharm when I "debug" it
doing the debug version

What is the easiest way to detect key presses in python 3 on a linux machine?

Right now I'm trying to make a small code with a raspberry pi and and a makey makey. The makey makey is a small board that acts as a usb keyboard when certain contacts are powered. My question is what is the easiest way to detect those keypresses inside a python script. I understand using the GPIO pins would be easier, but right now I'm looking for this. I have seen examples such as using using getch() from msvcrt (which from what I understand is windows only,) using pygame.key, and using getKey. Which of theses is easiest to use? Are there any that can detect a key being pressed and a key being released?
Pseudo code:
import whatever needs importing
if the "W" key is pressed:
print ("You pressed W")
elif the "S" is pressed:
print ("You pressed S")
and so on. Thanks.

This is a simple loop that will put stdin in raw mode (disabling buffering so you don't have to press enter) to get single characters. You should do something smarter (like a with statement to disable it) but you get the idea here:
import tty
import sys
import termios
orig_settings = termios.tcgetattr(sys.stdin)
tty.setcbreak(sys.stdin)
x = 0
while x != chr(27): # ESC
x=sys.stdin.read(1)[0]
print("You pressed", x)
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, orig_settings)
I think you'd have to loop to detect key releases in Python.
ETA some more explanation:
On Linux, input to your program will be line buffered. This means that the operating system will buffer up input until it has a whole line, so your program won't even see anything the user typed until the user also hits 'enter'. In other words, if your program is expecting the user to type 'w' and the user does this, 'w' will be sitting in the OS's buffer until the user hits 'enter'. At this point the entire line is delivered to your program so you will get the string "w\n" as the user's input.
You can disable this by putting the tty in raw mode. You do this with the Python function tty.setcbreak which will make a call down the tty driver in linux to tell it to stop buffering. I passed it the sys.stdin argument to tell it which stream I wanted to turn buffering off for1. So after the tty.setcbreak call, the loop above will give you output for every key the user presses.
A complication, though, is that once your program exits, the tty is still in raw mode. You'll generally find this unsatisfying since you don't get any of the power that modern terminal settings offer (like when you use control or escape sequences). For example, notice that you might have trouble exiting the program with ctrl-C. Consequently you should put the terminal back into cooked mode once you are done reading input characters. The termios.tcsetattr call simply says "put the terminal back the way I found it". It knows how to do this by first calling termios.tcgetattr at the beginning of the program which is saying "tell me all the current settings for the terminal".
Once you understand all that, you should easily be able to encapsulate the functionality in a function that suits your program.
1 stdin is the stream that input comes to you from the user. Wikipedia can tell you more about standard streams.

Using a good lightweight module curtsies you could do something like this (taken from their examples/ directory):
from curtsies import Input
def main():
with Input(keynames='curses') as input_generator:
for e in input_generator:
print(repr(e))
if __name__ == '__main__':
main()
So pressing keys on your keyboard gives you something like this:
'a'
's'
'KEY_F(1)'
'KEY_F(2)'
'KEY_F(3)'
'KEY_F(4)'
'KEY_F(5)'
'KEY_LEFT'
'KEY_DOWN'
'KEY_UP'
'KEY_RIGHT'
'KEY_NPAGE'
'\n'
curtsies is used by bpython as a low level abstraction of terminal-related stuff.
The basic problem of decoding the input is that in different terminals and terminal emulator programs like xterm or gnome-terminals physically same keys produce different keycode sequences. That's why one needs to know which terminal settings should be used to decode input. Such a module helps to abstract from those gory details.

Since your question states that you are using a Raspberry Pi and a USB HID keyboard peripheral, but does not specify whether or not you have the Pi configured to boot into text or graphical mode where you will be running your script, I would suggest using libinput which will work in either case.
You can use libinput's python bindings to read keyboard (and most other input devices) events directly from the kernel.
pip3 install python-libinput
The interface to this subsystem is exposed through the character devices which usually live in /dev/input/. They are managed by udev rules which create one or more character devices per attached input device, and are added and removed dynamically when, for example, a USB keyboard is attached or unplugged, or a Bluetooth mouse connects or disconnects.
Libinput handles for you the task of opening and reading from all attached input devices, and opening and closing devices when they are added and removed, respectively.
Using libinput from python to read key events would look like this:
import libinput
def read_key_events():
# init libinput
li = libinput.LibInput(udev=True)
li.udev_assign_seat('seat0')
# loop which reads events
for event in li.get_event():
# test the event.type to filter out only keyboard events
if event.type == libinput.constant.Event.KEYBOARD_KEY:
# get the details of the keyboard event
kbev = event.get_keyboard_event()
kcode = kbev.get_key() # constants in libinput.define.Key.KEY_xxx
kstate = kbev.get_key_state() # constants libinput.constant.KeyState.PRESSED or .RELEASED
# your key handling will look something like this...
if kstate == libinput.constant.KeyState.PRESSED:
print(f"Key {kcode} pressed")
elif kstate == libinput.constant.KeyState.RELEASED:
if kbev.get_key() == libinput.define.Key.KEY_ENTER:
print("Enter key released")
elif kcode == libinput.define.Key.KEY_SPACE:
print("Space bar released")
else:
print(f"Key {kcode} released")
One minor gotcha to be aware of is that udev is commonly configured to set permissions on the event devices it creates in /dev/input/ to allow access only from users who are members of a special supplementary 'input' group, since to allow unrestricted access to raw user key and mouse input would be a major security flaw. As such, o if running this throws an error during the libinput initialisation, you may need to add input to your user's supplementary groups by running:
sudo usermod -G input -a "${USERNAME}"

Executing commands at breakpoint in gdb via python interface

Not a duplicate of this question, as I'm working through the python interface to gdb.
This one is similar but does not have an answer.
I'm extending a gdb.breakpoint in python so that it writes certain registers to file, and then jumps to an address: at 0x4021ee, I want to write stuff to file, then jump to 0x4021f3
However, nothing in command is ever getting executed.
import gdb
class DebugPrintingBreakpoint(gdb.Breakpoint):
def __init__(self, spec, command):
super(DebugPrintingBreakpoint, self).__init__(spec, gdb.BP_BREAKPOINT, internal = False)
self.command = command
def stop(self):
with open('tracer', 'a') as f:
f.write(chr(gdb.parse_and_eval("$rbx") ^ 0x71))
f.close()
return False
gdb.execute("start")
DebugPrintingBreakpoint("*0x4021ee", "jump *0x4021f3")
gdb.execute("continue")
If I explicitly add gdb.execute(self.command) to the end of stop(), I get Python Exception <class 'gdb.error'> Cannot execute this command while the selected thread is running.:
Anyone have a working example of command lists with breakpoints in python gdb?

A couple options to try:
Use gdb.post_event from stop() to run the desired command later. I believe you'll need to return True from your function then call continue from your event.
Create a normal breakpoint and listen to events.stop to check if your breakpoint was hit.

The Breakpoint.stop method is called when, in gdb terms, the inferior is still "executing". Partly this is a bookkeeping oddity -- of course the inferior isn't really executing, it is stopped while gdb does a bit of breakpoint-related processing. Internally it is more like gdb hasn't yet decided to report the stop to other interested parties inside gdb. This funny state is what lets stop work so nicely vis a vis next and other execution commands.
Some commands in gdb can't be invoked while the inferior is running, like jump, as you've found.
One thing you could try -- I have never tried this and don't know if it would work -- would be to assign to the PC in your stop method. This might do the right thing; but of course you should know that the documentation warns against doing weird stuff like this.
Failing that I think the only approach is to fall back to using commands to attach the jump to the breakpoint. This has the drawback that it will interfere with next.
One final way would be to patch the running code to insert a jump or just a sequence of nops.

Python reading and writing to tty

BACKGROUND: If you want, skip to the problem section
I am working on a front end for test equipment. The purpose of the front end is to make it easier to write long test scripts. Pretty much just make them more human readable and writable.
The equipment will be tested using a Prologix GPIB-USB Controller (see prologix.biz). We found a tutorial at http://heliosoph.mit-links.info/gpib-on-debian-linux-the-easy-way/ and did all of the steps, and it worked!
As we don't have the test equipment yet, we wanted to write an emulator in Python using openpty. We do have the GPIB-USB Controller, just not what gets connected to that. I got the emulator working as a perfect replacement for the GPIB-USB. This means that I would follow the "GPIB on Debian ..." tutorial (above) and get output that I programmed the emulator to return. The input and output were done in the same manner as the tutorial just reading and writing to/from a pty device (ie /dev/pts/2) instead of the tty (ie /dev/ttyUSB0).
Now that the emulator works, we want to write a front end that can be used to write scripts easily. The goal is to make a kind of macro system that writes a bunch of commands when we call a function.
PROBLEM: exists using both the emulator and the device
I am using the following Python functions to read, write, and open the tty/pty devices, but I am not getting the same result that I get if I just use echo and cat in bash.
tty = os.open(tty_path, os.O_RDWR)
os.read(tty, 100)
os.write(tty, "++ver")
for example, I would expect the following to be equivalent
$ cat < /dev/pty/2 & # According to the tutorial, this must be run in parallel
$ echo "++ver" > /dev/pty/2
Prologix GPIB Version 1.2.3.4 ...
and
tty = os.open("/dev/pyt/2", os.o_RDWR)
os.read(tty, 100) # In separate Thread to be run in parallel
os.write(tty, "++ver") # in main thread
The output is very different, please explain why and how I can fix it.
FULL CODE is here: http://pastebin.com/PWVsMjD7

Well, I asked too soon. I hope someone benefits from this self answer.
So this works to read and write from both the emulator and the actual device. I am not exactly sure why, and would appreciate an explanation, but this does work in all of my tests
import serial
class VISA:
def __init__(self, tty_name):
self.ser = serial.Serial()
self.ser.port = tty_name
# If it breaks try the below
#self.serConf() # Uncomment lines here till it works
self.ser.open()
self.ser.flushInput()
self.ser.flushOutput()
self.addr = None
self.setAddress(0)
def cmd(self, cmd_str):
self.ser.write(cmd_str + "\n")
sleep(0.5)
return self.ser.readline()
def serConf(self):
self.ser.baudrate = 9600
self.ser.bytesize = serial.EIGHTBITS
self.ser.parity = serial.PARITY_NONE
self.ser.stopbits = serial.STOPBITS_ONE
self.ser.timeout = 0 # Non-Block reading
self.ser.xonxoff = False # Disable Software Flow Control
self.ser.rtscts = False # Disable (RTS/CTS) flow Control
self.ser.dsrdtr = False # Disable (DSR/DTR) flow Control
self.ser.writeTimeout = 2
def close(self):
self.ser.close()

You do not actually have to use any special module to read from TTY.
Option O_NOCTTY solved my problems with CDCACM example MCU app.
I'm sure it will work for you (as you work on Linux too).
#!/usr/bin/env python3
import io, os
tty = io.TextIOWrapper(
io.FileIO(
os.open(
"/dev/ttyACM1",
os.O_NOCTTY | os.O_RDWR),
"r+"))
for line in iter(tty.readline, None):
print(line.strip())

Stumbled on this while looking into pty/tty usage in python.
I think the original code did not work because echo will add a newline and the python os.write will not.
This is shown in your self answer here self.ser.write(cmd_str + "\n")
So the original code may have worked if it were os.write(tty, "++ver\n")