I would like to be able to gather the values for number of CPUs on a server and stuff like storage space etc and assign them to local variables in a python script. I have paramiko set up, so I can SSH to remote Linux nodes and run arbitrary commands on them, and then have the output returned to the script. However, many commands are very verbose "such as df -h", when all I want to assign is a single integer or value.
For the case of number of CPUs, there is Python functionality such as through the psutil module to get this value. Such as 'psutil.NUM_CPUS' which returns an integer. However, while I can run this locally, I can't exactly execute it on remote nodes as they don't have the python environment configured.
I am wondering how common it is to manually parse output of linux commands (such as df -h etc) and then grab an integer from it (similar to how bash has a "cut" function). Or whether it is somehow better to set up an environment on each remote server (or a better way).
Unfortunately it is very common to manually parse output of linux commands, but you shouln't. This is a really common server admin task and you shouldn't re invent the wheel.
You can use something like sar to log remote stats and retrieve the reports over ssh.
http://www.ibm.com/developerworks/aix/library/au-unix-perfmonsar.html
You should also look at salt. It lets you run the same command on multiple machines and get their output.
http://www.saltstack.com/
These are some of the options but remember to keep it DRY ;)
Like #Floris, I believe the simplest way is to design your commands so that the result is simple to parse. However, parsing the result of one command is not at all uncommon, bash scripts are full of grep, sed, wc or awk commands that do exactly that.
The same approach is used by psutils itself, see how it reads /proc/cpuinfo for cpu_count. You can implement the same parsing, only reading the distant /proc/cpuinfo, or counting the lines in the output of ls -1 /sys/bus/cpu/devices/.
Actually, the best way to get information will be from /proc and /sys, they are specially designed to ease access to internal information from simple programs, with minimal parsing needed.
If you can put your own programs or scripts on the remote machine there are a couple of things you can do:
Write a script on the remote machine that outputs just what you want, and execute that over ssh.
Use ssh to tunnel a port on the other machine and communicate with a server on the remote machine which will respond to requests for information with the data you want over a socket.
Since you already have an SSH connection I would suggest to wrap your commands with
Python's sh library. It's really nice for this kind of task and you get results really fast.
from sh import ssh
myserver = ssh.bake("myserver.com", p=1393)
print(myserver) # "/usr/bin/ssh myserver.com -p 1393"
# resolves to "/usr/bin/ssh myserver.com -p 1393 whoami"
iam2 = myserver.whoami()
Related
Preconditions:
I want to execute dyamic multiple commands via ssh from python on one remote machine at a time
I couldn't find any existing modules matching my "flavour" (If you care why, see below (*) ;))
Python scripts are running local on a Ubuntu machine
In general for single "one action calls" I simply do a native ssh call using subprocess.Popen and it works fine.
But for multiple subsequent dynamic calls, I don't want to create a new ssh connection for every command, even if the remote host might allow it. I thought of the following solution:
1) Configure my local ssh on Ubuntu to use multiplexing, so as long as a connection is open, it is used instead of creating a new one (https://www.admin-magazin.de/News/Tipps/Mit-SSH-Multiplexing-schneller-einloggen (Sorry, in german))
2) Creating an ssh connection by opening it in a running background thread, where in itself nothing is done, besides maybe a "keepalive" if necessary, or things like that, and keep the connection open till it's closed (i.e. by stopping the thread). (http://sebastiandahlgren.se/2014/06/27/running-a-method-as-a-background-thread-in-python/ )
3) Still executing ssh calls simply via subprocess.Popen, but now automatically using the open connection due to the ssh multiplexing config.
Should this work, or is there a fallacy alert?
(*)What I don't want:
Most solutions/examples I found used paramiko. On my first "happy path" it worked like charm, but the first failure test resulted in an internal AttributeError (https://github.com/paramiko/paramiko/issues/1617) and I don't want to build anything on this.
Other Libs i found like i.e. http://robotframework.org/SSHLibrary/SSHLibrary.html don't seem to have a real community using them.
pexpect....the whole "expect" concept gives me the creeps and should in my opinion only by used if there's absolutly no other reasonable reason ;)
What you've proposed is fine, but you don't even need to keep an ssh connection running in a background thread. If you configure ControlMaster (for reusing an existing connection) and ControlPerist (for keeping the master connection open even when all other connections have closed), then new ssh connections will continue to use the shared connection (as long as they happen before the ControlPersist timeout).
This means that if you set up the ControlMaster configuration external to your code (e.g., in ~/.ssh/ssh_config), your code doesn't even need to be aware of the configuration: it can just continue to call ssh normally, and ssh will take care of reusing the connection.
I have a python application("App1") that uses serial port /dev/ttyUSB0. This application is running as Linux service. It is running very well as it is perfectly automated for the task that I need it to perform. However, I have recently came to realize that sometimes I accidentally use the same serial port for another python application that I am developing which causes unwanted interference with "App1".
I did try to lock down "App1" as follows:
ser=serial.Serial(PORT, BAUDRATE)
fcntl.lockf(ser, fcntl.LOCK_EX | fcntl.LOCK_NB)
However, for other applications I sometimes unknowingly using
ser=serial.Serial(PORT, BAUDRATE)
without checking for ser.isOpen()
In order to prevent this I was wondering during the times I work on other applications, is there a way for ser=serial.Serial(PORT, BAUDRATE) to notify me that the serial port is already in use when I try to access it?
A solution I came up with is to create a cronjob that runs forever which essentially checks the following:
fuser -k /dev/ttyUSB0 #to get the PID of activated services that uses /dev/ttyUSB0
pkill -f <PID of second application shown in output above> #kill the application belonging to the second PID given by the above command
The above would ensure that whenever two applications use the same serial port, the second PID will get killed(I am aware there are some leaks in this logic). What do you guys think? If this is not a good solution, is there any way for ser=serial.Serial(PORT, BAUDRATE) to notify me that /dev/ttyUSB0 is already in use when I try to access it or do I need to implement the logic at driver level? Any help would be highly appreciated!
I think the simpler thing to do there is to create a different user for the "stable process", and use plain file permissions to give that user exclusive access to /dev/ttyUSB0.
With Unix groups and still plain permissions, that other process can still access any other resource it needs that would be under your main user - so it should be plain simple
If you are not aware of them, check the documentation for the commands chown and chmod.
I'm currently trying to automate a deployment process, involving 3 different machines :
userA#hostA, pwdA
userB#hostB, pwdB
userC#hostC, pwdC
Here's the scenario I would like to execute, using Python's Fabric library (to which I'm new) :
def deploy():
execute(taskA, hosts=[hostA])
execute(taskB, hosts=[hostB])
execute(taskC, hosts=[hostC])
I've tried to set the variable env.passwords like this:
env.passwords = {'userA#hostA:22':pwdA, 'userB#hostB:22':pwdB, 'userC#hostC:22':pwdC}
But it makes the SSH connection hanging.
My current workaround is to modify the variables env.user and env.password before each call to execute (I could also do that at the beginning of taskA, taskB and taskC). I really do not find that very elegant, and totally outside of Fabric's spirit.
If someone has ran into such situation, found him/herself with a hanging SSH connection while trying to use the env.passwords dict, I'm all yours ! And of course, if anyone already managed to make a more elegant multi-host-password handling, I would be glad to hear about any hint.
It might be better to keep your ssh details in ~/.ssh/config and have Fabric use that. See
Leveraging native SSH config fileshttp://docs.fabfile.org/en/1.4.0/usage/execution.html#ssh-config
Connecting to a host listed in ~/.ssh/config when using Fabrichttps://stackoverflow.com/a/9685171
As said in my comment above, using the use_ssh_config variable works well. But in case you SSH with a password, Fabric currently doesn't handle per-host/per-user/per-password connection (which is normal, since using SSH passwords is considered as insecure)
That's why I dug into Fabric's source code, and added that new "feature". Here is my repository if you want to have a look. Basically, I used the env.roledefs variable, which lists roles as dictionary entries. And each role has a list of hosts, a user, and a password. Now, if you want to execute a task in one or more hosts (which have different usernames and passwords), there are two ways to do it:
Annotating the task with the decorator #complete_roles('role1',
'role2')
Using the function execute(task, complete_roles=['role1', 'role2'])
Just be sure to have 'role1' and 'role2' as keys in env.roledefs
If you want to have more info about my contribution, just check the two latest commits. I know, my code is not the cleanest...
I've used web.py to create a web service that returns results in json.
I run it on my local box as python scriptname.py 8888
However, I now want to run it on a linux box.
How can I run it as a service on the linux box?
update
After the answers it seems like the question isn't right. I am aware of the deployment process, frameworks, and the webserver. Maybe the following back story will help:
I had a small python script that takes as input a file and based on some logic splits that file up. I wanted to use this script with a web front end I already have in place (Grails). I wanted to call this from the grails application but did not want to do it by executing a command line. So I wrapped the python script as a webservice. which takes in two parameters and returns, in json, the number of split files. This webservice will ONLY be used by my grails front end and nothing else.
So, I simply wish to run this little web.py service so that it can respond to my grails front end.
Please correct me if I'm wrong, but would I still need ngix and the like after the above? This script sounds trivial but eventually i will be adding more logic to it so I wanted it as a webservice which can be consumed by a web front end.
In general, there are two parts of this.
The "remote and event-based" part: Service used remotely over network needs certain set of skills: to be able to accept (multiple) connections, read requests, process, reply, speak at least basic TCP/HTTP, handle dead connections, and if it's more than small private LAN, it needs to be robust (think DoS) and maybe also perform some kind of authentication.
If your script is willing to take care of all of this, then it's ready to open its own port and listen. I'm not sure if web.py provides all of these facilities.
Then there's the other part, "daemonization", when you want to run the server unattended: running at boot, running under the right user, not blocking your parent (ssh, init script or whatever), not having ttys open but maybe logging somewhere...
Servers like nginx and Apache are built for this, and provide interfaces like mod_python or WSGI, so that much simpler applications can give up as much of the above as possible.
So the answer would be: yes, you still need Nginx or the likes, unless:
you can implement it yourself in Python,
or you are using the script on localhost only and are willing to take some
risks of instability.
Then probably you can do on your own.
try this
python scriptname.py 8888 2>/dev/null
it will run as daemon
I wrote the code like this
import smtplib
server=smtplib.SMTP('localhost')
Then it raised an error like
error: [Errno 10061] No connection could be made because the target machine actively refused it
I am new to SMTP, can you tell what exactly the problem is?
It sounds like SMTP is not set up on the computer you are trying this from. Try using your ISP's mail server (often something like mail.example.com) or make sure you have an SMTP server installed locally.
Rather than trying to install smtp library locally, you can setup a simple smtp server on a console.
Do this:
python -m smtpd -n -c DebuggingServer localhost:1025
And all mails will be printed to the console.
To send e-mail using the Python SMTP module you need to somehow obtain the name of a valid mail exchanger (MX). That's either a local hub or smart relay of your own or you can query DNS for the public MX records for each target host/domain name.
This requirement is glossed over in the docs. It's a horrible omission in Python's standard libraries is that they don't provide an easy way to query DNS for an MX record. (There are rather nice third party Python DNS libraries such as DNSPython and PyDNS with extensive support for way more DNS than you need for anything related to e-mail).
In general you're probably better off using a list of hubs or relays from your own network (or ISP). This is because your efforts to send mail directly to the published MX hosts may otherwise run afoul of various attempts to fight spam. (For example it frequently won't be possible from wireless networks in coffee shops, hotels, and across common household cable and DSL connections; most of those address ranges are listed in various databases as potential sorts of spam). In that case you could store and/or retrieve the names/address of your local mail hubs (or smart relays) through any means you like. It could be a .cfg file (ConfigParser), or through an LDAP or SQL query or even (gasp!) hard-coded into your scripts.
If, however, your code is intended to run on a suitable network (for example in a colo, or a data center) then you'll have to do your own MX resolution. You could install one of the aforementioned PyPI packages. If you need to limit yourself to the standard libraries then you might be able to rely on the commonly available dig utility that's included with most installations of Linux, MacOS X, Solaris, FreeBSD and other fine operating systems.
In that case you'd call a command like dig +short aol.com mx | awk '{print $NF}' through subprocess.Popen() which can be done with this rather ugly one-liner:
mxers = subprocess.Popen("dig +short %s mx | awk '{print $NF}'"
% target_domain, stdout=subprocess.PIPE,
shell=True).communicate()[0].split()
Then you can attempt to make an SMTP connection to each of the resulting hostnames in
turn. (This is fine so long as your "target_domain" value is adequately sanitized; don't pass untrusted data through Popen() with shell=True).
The safer version looks even hairier:
mxers = subprocess.Popen(["dig", "+short", target_domain, "mx"],
stdout=subprocess.PIPE).communicate()[0].split()[1::2]
... where the slice/stride at the end is replacing the call to awk and thus obviating
the need for shell=True