I bought this temperature sensor logger kit: http://quozl.netrek.org/ts/. It works great with the supplied C code, I like to use python because of its simplicity, so I wrote a script in python that displays the output from the microcontroller. I only have one temperature sensor hooked up to the kit. I want the temperature to be displayed on a web page, but can't seem to figure it out, I'm pretty sure it has something to do with the output from the micro having a \r\n DOS EOL character and linux web servers do not interpret it properly. The book I have says "Depending on the web server you are using, you might need to make configuration changes to understand how to serve CGI files." I am using debian and apache2 and basic cgi scripts work fine.
Here is my code for just displaying the sensor to the console (this works fine):
import serial
ser = serial.Serial('/dev/ttyS0', 2400)
while 1:
result = ser.readline()
if result:
print result
Here is my test.cgi script that works:
#!/usr/bin/python
print "Content-type: text/html\n"
print "<title>CGI Text</title>\n"
print "<h1>cgi works!</h1>"
Here is the cgi script I have started to display temp (doesn't work - 500 internal server error):
#!/usr/bin/python
import sys, serial
sys.stderr = sys.stdout
ser = serial.Serial('/dev/ttyS0', 2400)
print "Content-type: text/html\n"
print """
<title>Real Time Temperature</title>
<h1>Real Time Temperature:</h1>
"""
#result = ser.readline()
#if result:
print ser.readline()
If i run python rtt.cgi in the console it outputs the correct html and temperature, I know this will not be real time and that the page will have to be reloaded every time that the user wants to see the temperature, but that stuff is coming in the future.. From my apache2 error log it says:
malformed header from script. Bad header= File "/usr/lib/cgi-bin/rtt.c: rtt.cgi
I'm guessing that the execution context under which your CGI is running is unable to complete the read() from the serial port.
Incidentally the Python standard libraries have MUCH better ways for writing CGI scripts than what you're doing here; and even the basic string handling offers a better way to interpolate your results (assuming you code has the necessary permissions to read() them) into the HTML.
At least I'd recommend something like:
#!/usr/bin/python
import sys, serial
sys.stderr = sys.stdout
ser = serial.Serial('/dev/ttyS0', 2400)
html = """Content-type: text/html
<html><head><title>Real Time Temperature</title></head><body>
<h1>Real Time Temperature:</h1>
<p>%s</p>
</body></html>
""" % ser.readline() # should be cgi.escape(ser.readline())!
ser.close()
sys.exit(0)
Notice we just interpolate the results of ser.readline() into our string using the
% string operator. (Incidentally your HTML was missing <html>, <head>, <body>, and <p> (paragraph) tags).
There are still problems with this. For example we really should at least import cgi wrap the foreign data in that to ensure that HTML entities are properly substituted for any reserved characters, etc).
I'd suggest further reading: [Python Docs]: http://docs.python.org/library/cgi.html
one more time:
# Added to allow cgi-bin to execute cgi, python and perl scripts
ScriptAlias /cgi-bin/ /var/www/cgi-bin/
AddHandler cgi-script .cgi .py .pl
<Directory /var/www>
Options +Execcgi
AddHandler cgi-script .cgi .py .pl
</Directory>
Michael,
It looks like the issue is definitely permissions, however, you shouldn't try to make your script have the permission of /dev/ttyS0. What you will probably need to do is spawn another process where the first thing you do is change your group to the group of the /dev/ttyS0 device. On my box that's 'dialout' you're may be different.
You'll need to import the os package, look in the docs for the Process Parameters, on that page you will find some functions that allow you to change your ownership. You will also need to use one of the functions in Process Management also in the os package, these functions spawn processes, but you will need to choose one that will return the data from the spawned process. The subprocess package may be better for this.
The reason you need to spawn another process is that the CGI script need to run under the Apache process and the spawn process needs to access the serial port.
If I get a chance in the next few days I'll try to put something together for you, but give it a try, don't wait for me.
Also one other thing all HTTP headers need to end in two CRLF sequences. So your header needs to be:
print "Content-type: text/html\r\n\r\n"
If you don't do this your browser may not know when the header ends and the entity data begins. Read RFC-2616
~Carl
Related
As an example I am trying to "imitate" the behaviour of the following sets of commands is bash:
mkfifo named_pipe
/challenge/embryoio_level103 < named_pipe &
cat > named_pipe
In Python I have tried the following commands:
import os
import subprocess as sp
os.mkfifo("named_pipe",0777) #equivalent to mkfifo in bash..
fw = open("named_pipe",'w')
#at this point the system hangs...
My idea it was to use subprocess.Popen and redirect stdout to fw...
next open named_pipe for reading and giving it as input to cat (still using Popen).
I know it is a simple (and rather stupid) example, but I can't manage to make it work..
How would you implement such simple scenario?
Hello fellow pwn college user! I just solved this level :)
open(path, flags) blocks execution. There are many similar stackoverflow Q&As, but I'll reiterate here. A pipe will not pass data until both ends are opened, which is why the process hangs (only 1 end was opened).
If you want to open without blocking, you may do so on certain operating systems (Unix works, Windows doesn't as far as I'm aware) using os.open with the flag os.O_NONBLOCK. I don't know what consequences there are, but be cautious of opening with nonblocking because you may try reading prematurely and there will be nothing to read (possibly leading to error, etc.).
Also, note that using the integer literal 0777 causes a syntax error, so I assume you mean 0o777 (max permissions), where the preceding 0o indicates octal. The default for os.mkfifo is 0o666, which is identical to 0o777 except for the execute flags, which are useless because pipes cannot be executed. Also, be aware that these permissions might not all be granted and when trying to set to 0o666, the permissions may actually be 0o644 (like in my case). I believe this is due to the umask, which can be changed and is used simply for security purposes, but more info can be found elsewhere.
For the blocking case, you can use the package multiprocessing like so:
import os
import subprocess as sp
from multiprocessing import Process
path='named_pipe'
os.mkfifo(path)
def read(): sp.run("cat", stdin=open(path, "r"))
def write(): sp.run(["echo", "hello world"], stdout=open(path, "w"))
if __name__ == "__main__":
p_read = Process(target=read)
p_write = Process(target=write)
p_read.start()
p_write.start()
p_read.join()
p_write.join()
os.remove(path)
output:
hello world
In the past week I install a Terraria 1.3.5.3 server into an Ubuntu v18.04 OS, for playing online with friends. This server should be powered on 24/7, without any GUI, only been accessed by SSH on internal LAN.
My friends ask me if there is a way for them to control the server, e.g. send a message, via internal in-game chat, so I thought use a special character ($) in front of the desired command ('$say something' or '$save', for instance) and a python program, that read the terminal output via pipe, interpreter the command and send it back with a bash command.
I follow these instructions to install the server:
https://www.linode.com/docs/game-servers/host-a-terraria-server-on-your-linode
And config my router to forward a dedicated port to the terraria server.
All is working fine, but I really struggle to make python send a command via "terrariad" bash script, described in the link above.
Here is a code used to send a command, in python:
import subprocess
subprocess.Popen("terrariad save", shell=True)
This works fine, but if I try to input a string with space:
import subprocess
subprocess.Popen("terrariad \"say something\"", shell=True)
it stop the command in the space char, output this on the terminal:
: say
Instead of the desired:
: say something
<Server>something
What could I do to solve this problem?
I tried so much things but I get the same result.
P.S. If I send the command manually in the ssh putty terminal, it works!
Edit 1:
I abandoned the python solution, by now I'll try it with bash instead, seem to be more logic to do this way.
Edit 2:
I found the "terrariad" script expect just one argument, but the Popen is splitting my argument into two no matter the method I use, as my input string has one space char in the middle. Like this:
Expected:
terrariad "say\ something"
$1 = "say something"
But I get this of python Popen:
subprocess.Popen("terrariad \"say something\"", shell=True)
$1 = "say
$2 = something"
No matter i try to list it:
subprocess.Popen(["terrariad", "say something"])
$1 = "say
$2 = something"
Or use \ quote before the space char, It always split variables if it reach a space char.
Edit 3:
Looking in the bash script I could understand what is going on when I send a command... Basically it use the command "stuff", from the screen program, to send characters to the terraria screen session:
screen -S terraria -X stuff $send
$send is a printf command:
send="`printf \"$*\r\"`"
And it seems to me that if I run the bash file from Python, it has a different result than running from the command line. How this is possible? Is this a bug or bad implementation of the function?
Thanks!
I finally come with a solution to this, using pipes instead of the Popen solution.
It seems to me that Popen isn't the best solution to run bash scripts, as described in How to do multiple arguments with Python Popen?, the link that SiHa send in the comments (Thanks!):
"However, using Python as a wrapper for many system commands is not really a good idea. At the very least, you should be breaking up your commands into separate Popens, so that non-zero exits can be handled adequately. In reality, this script seems like it'd be much better suited as a shell script.".
So I came with the solution, using a fifo file:
First, create a fifo to be use as a pipe, in the desired directory (for instance, /samba/terraria/config):
mkfifo cmdOutput
*/samba/terraria - this is the directory I create in order to easily edit the scripts, save and load maps to the server using another computer, that are shared with samba (https://linuxize.com/post/how-to-install-and-configure-samba-on-ubuntu-18-04/)
Then I create a python script to read from the screen output and then write to a pipe file (I know, probably there is other ways to this):
import shlex, os
outputFile = os.open("/samba/terraria/config/cmdOutput", os.O_WRONLY )
print("python script has started!")
while 1:
line = input()
print(line)
cmdPosition = line.find("&")
if( cmdPosition != -1 ):
cmd = slice(cmdPosition+1,len(line))
cmdText = line[cmd]
os.write(outputFile, bytes( cmdText + "\r\r", 'utf-8'))
os.write(outputFile, bytes("say Command executed!!!\r\r", 'utf-8'))
Then I edit the terraria.service file to call this script, piped from terrariaServer, and redirect the errors to another file:
ExecStart=/usr/bin/screen -dmS terraria /bin/bash -c "/opt/terraria/TerrariaServer.bin.x86_64 -config /samba/terraria/config/serverconfig.txt < /samba/terraria/config/cmdOutput 2>/samba/terraria/config/errorLog.txt | python3 /samba/terraria/scripts/allowCommands.py"
*/samba/terraria/scripts/allowCommands.py - where my script is.
**/samba/terraria/config/errorLog.txt - save Log of errors in a file.
Now I can send commands, like 'noon' or 'dawn' so I can change the in-game time, save world and backup it with samba server before boss fights, do another stuff if I have some time XD, and have the terminal showing what is going on with the server.
Due to Apache gateway timeouts, and a desire to display more information to the end user, I'd like to be able to flush STDOUT on a python CGI script hosted on PCF, essentially giving updates on the status of the script.
I have tried enabling the -u tag in python (#!/usr/python -u at head of my script), sys.stdout.flush() command, and even using subprocess.call to execute a perl script that is set to flush to STDOUT that prints some progress text ($| = 1; at beginning of perl script). Furthermore, I've double checked that I'm not using any Apache modules that would require buffering (no mod_deflate, for example). Finally, I'll mention that executing a standard perl CGI rather than a python CGI allows for the STDOUT flushing, so I figure it must be something with my python/Apache/PCF configuration.
I'm fresh out of ideas here, and would like some advice.
With any of these above methods, I would have thought stdout would flush. But, none of them have worked!
Thanks in advance for any assisstance.
You can disable buffering using something like this in your Python2 code:
# set stdout as non-buffered
if hasattr(sys.stdout, 'fileno'):
fileno = sys.stdout.fileno()
tmp_fd = os.dup(fileno)
sys.stdout.close()
os.dup2(tmp_fd, fileno)
os.close(tmp_fd)
sys.stdout = os.fdopen(fileno, "w", 0)
That is reopening sys.stdout with no buffer (i.e. the 0 as third arg). After you do that, anything writing to sys.stdout should not be buffered.
For my A level computing project for year 13 im writing an email client, I need to Model how pythons SMTP protocol works and show protocol handshaking. What I want to know is that when I log into gmails mail server to send an email using smtp is there a way to print out what the line of code does.
So I would want to show exactly what is going on when the line is executed.
import smtplib
server = smtplib.SMTP('smtp.gmail.com', 587)
server.login("youremailusername", "password")
msg = "\nHello!" # The /n separates the message from the headers
server.sendmail("you#gmail.com", "target#example.com", msg)
Cheers guys
Assuming that by "what the line of code does" you mean "what protocol messages get sent to and received from the server", smtplib.SMTP.set_debuglevel(level):
Set the debug output level. A true value for level results in debug messages for connection and for all messages sent to and received from the server.
If by "what the line of code does" you want to know the Python code that's being executed, you can step into the function call in the debugger. Or just read the source. Like many modules in the stdlib, smtplib is designed to be useful as sample code as well as a practical module, so at the top of the docs, there's a link to smtplib.py.
Is there a way I can write that output to a tkinter window or file?
If you look at the source linked above, you can see that it just uses print calls for its debug logging. So, this gives you a few options:
Fork smtplib and replace those print calls with something better.
Monkeypatch smtplib to give it a print function that shadows the global one. (This only works in Python 3.x; in 2.x, print isn't a function.)
Open a text file, and just assign sys.stderr = my_text_file. (This only works for files, not tkinter. And it also catches all stderr, not just the logging from smtplib.)
Create a file-like object that does whatever you want in its write method, and assign sys.stderr to that. (This works for anything you want to do, including adding to a tkinter edit window, but of course it still catches all stderr.)
Wrap the script from outside—e.g., with a wrapper script that uses subprocess.Popen to call the real script and capture its stderr in a pipe.
Which one is appropriate depends on your needs. For your assignment, assuming nothing is writing to stderr but smtplib's debug output during the time you're doing smtplib stuff, I think the file-like object idea might make sense. So:
class MyDumbFakeStderr(object):
def write(self, output):
add_to_my_file_or_tkinter_thing(output)
sys.stderr = MyDumbFakeStderr()
try:
# your smtplib code here
finally:
sys.stderr = sys.__stderr__
Obviously restoring stderr is unnecessary if you're just going to quit as soon as you're done, while if you want to do it repeatedly you'll probably want to wrap it in a contextlib.contextmanager, etc. Also, this MyDumbFakeStderr is pretty dumb (hence the name); it works fine for wrapping code that does nothing but print whole lines to stderr, but for anything more complicated you might need to add your own line buffering, or make it an io.TextIOBase, etc. This is just an idea to get you started.
You can read the function's source code.
http://www.opensource.apple.com/source/python/python-3/python/Lib/smtplib.py (search for sendmail)
You can also read a bit about SMTP: http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#SMTP_transport_example
And try to relate the two
I am using Cherrypy framework to run my python code on server. But the process stops working when the load increases.
Every time this happens I have to manually go and start the python code. Is there any way i can use Gunicorn with Cherrypy so that Gunicorn can start the code automatically when it stops working.
Any other solution will also work in this case. Just want to make sure that the python program does not stop working.
I use a cron that checks the memory load every few minutes and resets cherrypy when the memory exceeds 500MB -- so that the web host doesn't complain to me with emails. Something on my server doesn't release memory when a function ends as it should, so this is a pragmatic work around.
This hack may be weird because I reset it using an HTTP request, but that's because I spent hours trying to figure out how to do this withing the BASH and gave up. It works.
CRON PART
*/2 * * * * /usr/local/bin/python2.7 /home/{mypath}/cron_reset_cp.py > $HOME/cron.log 2>&1
And code inside cron_reset_cp.py...
#cron for resetting cherrypy /cp/ when 500+ MB
import os
#assuming starts in /home/my_username/
os.chdir('/home/my_username/cp/')
import mem
C = mem.MemoryMonitor('my_username') #this function adds up all the memory
memory = int(float(C.usage()))
if memory > 500:#MB
#### Tried: pid = os.getpid() #current process = cronjob --- THIS approach did not work for me.
import urllib2
cp = urllib2.urlopen('http://myserver.com/cp?reset={password}')
Then I added this function to reset the cherrypy via cron OR after a github update from any browser (assuming only I know the {password})
The reset url would be http://myserver.com/cp?reset={password}
def index(self, **kw):
if kw.get('reset') == '{password}':
cherrypy.engine.restart()
ip = cherrypy.request.headers["X-Forwarded-For"] #get_client_ip
return 'CherryPy RESETTING for duty, sir! requested by '+str(ip)
The MemoryMonitor part is from here:
How to get current CPU and RAM usage in Python?
Python uses many error handling strategies to control flow. A simple try/except statement could throw an exception if, say, your memory overflowed, a load increased, or any number of issues making your code stall (hard to see without the actual code).
In the except clause, you could clear any memory you allocated and restart your processes again.
Depending on your OS, try the following logic:
Implement a os.pid() > /path/pid.file
Create a service script that connected to your web-port
Try to fetch data
If no data was recieved, kill PID #/path/pid.file
restart script
Your main script:
import os
with open('./pidfile.pid', 'wb') as fh:
fh.write(str(os.getpid()))
... Execute your code as per normal ...
service.py script:
from socket import *
from os import kill
s = socket()
try:
s.connect(('127.0.0.1', 80))
s.send('GET / HTTP/1.1\r\n\r\n')
len = len(s.recv(8192))
s.close()
except:
len = 0
if len <= 0:
with open('/path/to/pidfile.pid', 'rb') as fh:
kill(int(fh.read()))
And have a cronjob (execute in a console):
sudo export EDITOR=nano; crontab -e
Now you're in the text-editor editing your cronjobs, write the following two lines at the bottom:
*/2 * * * * cd /path/to/service.py; python service.py
*/5 * * * * cd /path/to/main/script.py; python script.py
Press Ctrl+X and when asked to save changes, write Y and enjoy.
Also, instead of restarting your script from within service.py script, i'd suggest that service.py kills the PID located in /path/pid.file and let your OS handle starting up your script if the PID is missing in /path/, Linux at least have very nifty features for this.
Best practice Ubuntu
It's more than considered best practice to use the systems service status apache2 for instance, the service scripts lets you reload, stop, start, get job states and what not.
Check out: http://upstart.ubuntu.com/cookbook/#service-job
And check the service scripts for all the other applications and not only use them as skeletons but make sure your application follows the same logic.
Perhaps you need supervisord to monitor your gunicorn process and restart it when it's necessary:
http://www.onurguzel.com/managing-gunicorn-processes-with-supervisor/