Develop Reference

Running a python script which uses paramiko to ssh into other server and perform db refresh from jenkins

I have a python script that does some Database operations from an ec2 server by SSHing into another server using paramiko. The script runs fine when I run it directly from the server as ec2-user but when I run the same from Jenkins I get a permission error on /home/ec2-user/.ssh/id_rsa file.
used python3.8 /home/ec2-user/db_refresh.py command to run the script from Jenkins
After some reading and with the help of whomai command, I found that's expected since Jenkins runs the scripts as Jenkins user and no one part from the owner has permissions to read private keys in ~/.ssh/ folder.
I could change the permission so that everyone can read ec2-user's private key but I think that would be a terrible idea(As far as I've read) and I think ssh wouldn't even work if anyone apart from the owner has read permission to that private key(I remember reading it somewhere but not sure)
sshcon = paramiko.SSHClient()
sshcon.connect(MYSQL_HOST, username=SSH_USERNAME, key_filename='/home/ec2-user/.ssh/id_rsa')
That is how SSH into my database server using paramiko.
Can I run my scripts from jenkins as ec2-user or is there some other way that I can overcome this.

In the end, it turned to be quite simple(stupid me)
I just created a key pair for Jenkins user and used it for doing my operations.
One thing to note is since jenkins is service account normal su jenkins won't work. I had to do this sudo su -s /bin/bash jenkins.

How can one use the command-line to use openvpn on windows?

I have been trying to come up with a way to interact with openvpn on windows.
I did fine the openvpn.exe but there are no commands that will use a config file to connect with a vpn server.
I am using nordvpn by the way. Specifically a tcp Canadian server.
I did find some people suggesting to use the openvpn-gui.exe to run the program but it defeats the purpose. I want t to completely be on command line.
Secondly, i need the command line to work as i want to integrate openvopn into a script which will download stuff from the internet.
Anyone with any idea how to do it?

For anyone looking into this in 2021, please find below a connect and a disconnect.
To connect
"C:\Program Files\OpenVPN\bin\openvpn-gui.exe" --command connect yourconfigfile.ovpn
To disconnect
"C:\Program Files\OpenVPN\bin\openvpn-gui.exe" --command disconnect yourconfigfile.ovpn
For windows users...
Both of these can be placed in a .bat file and automated with task scheduler, works like a charm.

Another CMD example:
"C:\Program Files\OpenVPN\bin\openvpn-gui.exe" --connect config.ovpn
Replace 'config.ovpn' by your configuration file. You also may have another path to openvpn-gui.exe.
It will work if you have no openVPN correctly installed so you can't run openvpn-gui ... from cmd.

You can actually use the config file with openvpn.exe, but you must provide the full path to it unlike with openvpn-gui.exe:
openvpn.exe --config "path/to/config.ovpn"

So I figured out the answer to the problem.
To control openvpn from command-line on windows:
Set the Environment Variable so you can access the openvpn-gui.exe from command-line.
Type this command: openvpn-gui --connect [config file]. (You are going to need the config file so download it from your vpn provider)
If you don't want to type the password every time you run the script, edit the config file to read the username and password from a .txt file. Here's a link that can help you do that: https://help.vpntunnel.com/support/solutions/articles/5000613671-how-do-i-save-my-username-password-in-openvpn-for-automatic-login-

Instructions
Step 1. Go to the correct location for x64 systems:
cd "%ProgramFiles%\OpenVPN Connect"
Note: if you have the 32 bits program installed on a 64 bits OS, replace %ProgramFiles% with %ProgramFiles(x86)%.
Step 2. Install the system service:
ovpnconnector.exe install
Step 3. Specify connection profile to use (optional):
ovpnconnector.exe set-config profile <FULL_PATH_AND_FILENAME_TO_PROFILE.OVPN>
Note: if your OpenVPN Connect installation file was downloaded from Access Server or OpenVPN Cloud and came with a bundled autologin connection profile, then you can skip step 3. It will then simply default to the bundled connection profile. It can be found in the program location with the name "ovpnconnector.ovpn" - that is the bundled connection profile.
Step 4. Specify the path to a log file (optional):
ovpnconnector.exe set-config log <FULL_PATH_AND_FILENAME_TO_LOGFILE.LOG>
Note: if you skip step 4, the service will write to the default log file in the program location with the name “ovpnconnector.log”.
Step 5. Start the service:
ovpnconnector.exe start
The service will now start the VPN connection and log output to the log file.
Note: you will not receive feedback after starting the service if the connection succeeded or not. You can check the log file or use the ping command to verify that the connection is now up and running.
Important: OpenVPN Connect client should not be running, otherwise service startup will abort.

How to login inside a AWS server & do some maintenance on it using Python?

I need to login inside a AWS Linux server, then create a folder, add some ownership on it and lastly restart tomcat.
I know that I should be using Ansible or any config mgmt tool & that's easy way.. but out of curiosity I want to do it using Python.
So basically, the steps that need to be followed are:
Login to Machine
mkdir /mnt/some_new_folder
Give permissions, chown tomcat7:tomcat7 /mnt/some_new_folder
Lastly restart tomcat: sudo service tomcat7 restart
Lastly logout
Is it possible to do all this via Python script ?

With open source tools like Python everything is possible. Only your knowledge sets the limit.
I would suggest using sh module which allows easy execution of remote commands over SSH.
sh + SSH tutorial.
You can use it like:
import sh
print(sh.ssh("username#example.com", "mkdir /foo/bar"))
First you need to setup proper SSH keys and SSH agent.

How can you automate terminal commands?

I'm tired of doing this.
ssh me#somehost.com
input my password
sudo su - someuser
input my password
cd /some/working/directory
<run some commands>
Is there anyway to automate this? Do I need a special shell? or a shell emulator? can I programmatically drive the shell up to certain point then run manual commands on it?
Bonus points of it's programmed in python for extra hacking goodness
edit: All the answers below focus on the "full automation" part of the question: Where the hard part is what I highlighted above. Here is another example to see if I can capture the essence.
ssh me#somehost.com
<get a shell because keys are setup>
sudo su - user_that_deploys_the_app
<input password, because we don't want to give passwordless sudo to developers>
cd env; source bin/activate
cd /path/where/ur/app/is/staging
<edit some files, restart the server, edit some more, check the logs, etc.>
exit the term

For the ssh/authentication piece, you can setup passwordless authentication by using keys. Then you can simply use ssh and a bash script to execute a series of commands in an automated fashion.
You could use Python here, but if you are executing a series of shell commands, it's probably a better idea to use a shell script, as that's precisely what they do.
Alternately, look into Fabric for your automation needs. It's Python-based, and your "recipes" are written in Python.

I'm not quite sure what you're asking, but what you're probably asking about is getting SSH working in password-less mode using public keys. The general idea is you generate an SSH keypair:
ssh-keygen -t rsa
which gives you id_rsa and id_rsa.pub. You append the contents of id_rsa.pub to the ~/.ssh/authorized_keys file of your target user, and SSH from that point on will not ask for credentials. In your example, this will work out to:
Only once
# On your source machine
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub
# Copy this to clip board
# On somehost.com
su - someuser
# edit ~/.ssh/authorized_keys and paste what you had copied from previous step
From now on, you can now just run
ssh someuser#somehost.com "sh -c 'cd /some/dir; command.sh'"
and not be prompted for credentials.

fabric is a fine choice, as others have pointed out. there is also pexpect which might be more what you're looking for.

You can play with autoexpect. It creates expect script (script language intended to handle interaction with user). Run
autoexpect ssh me#somehost.com
followed by rest of commands. Script script.exp will be created.
Please note, that exact results of input and output will be recorded by the script. If output may differ from execution to execution, you'll need to modify a bit generated script.

As Daniel pointed out you need to have a secure way of doing ssh and sudo on the boxes. Those items are universal to dealing with linux/unix boxes. Once you've tackled that you can use fabric. It's a python based tool to do automation.

You can set stuff up in your ~/.ssh/config
For example:
Host somehost
User test
See ssh_config(5) for more info.
Next, you can generate a SSH key using ssh-keygen(1), run ssh-agent(1), and use that for authentication.
If you want to run a command on a remote machine, you can just use something like:
$ ssh somehost "sh myscript.sh ${myparameter}".
I hope this at least points you in the right direction :)
If you need sudo access, then there are obvious potential security issues though ... You can use ChrootDirectory on a per user basis inside a Match block though. See sshd_config(5) for info.

try module paramiko. This can meet your requirement.

Cannot write a script to "svn export" in Python

I would like to write a script that will tell another server to SVN export a SVN repository.
This is my python script:
import os
# svn export to crawlers
for s in ['work1.main','work2.main']:
cmd = 'ssh %s "cd /home/zes/ ; svn --force export svn+ssh://174.113.224.177/home/svn/dragon-repos"' % s
print cmd
os.system(cmd)
Very simple. It will ssh into work1.main, then cd to a correct directory. Then call SVN export command.
However, when I run this script...
$ python export_to_crawlers.py
ssh work1.main "cd /home/zes/ ; svn --force export svn+ssh://174.113.224.177/home/svn/dragon-repos"
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-with-mic,password).
svn: Connection closed unexpectedly
ssh work2.main "cd /home/zes/ ; svn --force export svn+ssh://174.113.224.177/home/svn/dragon-repos"
Host key verification failed.
svn: Connection closed unexpectedly
Why do I get this error and cannot export the directory? I can manually type the commands in the command line and it will work. Why can't it work in the script?
If I change to this...it will not work. and instead, nothing will happen.
cmd = 'ssh %s "cd /home/zes/ ;"' % s

This is a problem with SSH.
Permission denied, please try again.
This means that ssh can't login. Either your ssh agent doesn't have the correct key loaded, you're running the script as a different user or the environment isn't passed on correctly. Check that the variables SSH_AUTH_SOCK and SSH_AGENT_PID are passed to the subprocess of your python script.
Host key verification failed.
This error means that the remote host isn't known to ssh. This means that the host key is not found in the file $HOME/.ssh/known_hosts. Again, make sure that you're checking the home directory of the effective user of the script.
[EDIT] When you run the script, then python will become the "input" of ssh: ssh is no longer connected to a console and will ask python for the password to login. Since python has no idea what ssh wants, it ignores the request. ssh tries three times and dies.
To solve it, run these commands before you run the Python script:
eval $(ssh-agent)
ssh-add path-to-your-private-key
Replace path-to-your-private-key with the path to your private key (the one which you use to login). ssh-add will ask for your password and the ssh-agent will save it in a secure place. It will also modify your environment. So when SSH runs the next time, it will notice that an ssh agent is running and ask it first. Since the ssh-agent knows the password, ssh will login without bothering Python.
To solve the second issue, run the second ssh command manually once. ssh will then add the second host to its files and won't ask again.
[EDIT2] See this howto for a detailed explanation how to login on a remote server via ssh with your private key.

I guess that it is related to ssh. Are you using a public key to automatically connect. I think that your shell knows this key but it is not the case of python.
I am not sure but it's just an idea. I hope it helps

Check out the pxssh module that is part of the pyexpect project:
https://pexpect.readthedocs.org/en/latest/api/pxssh.html
It simplifies dealing with automating ssh-ing into machines.