Develop Reference

Setting up crontab for my Django Application

I had an issue with setting up crontab for my Django application for a week and I have almost figured out the same. (Issue linked with Unable to make a function call using Django-cron)
My crontab -e syntax is
* * * * * /Users/ashwin/dashboard/proj_application/exec.sh >> /Users/ashwin/dashboard/proj_application/data.log 2>&1
And in my exec.sh, I have
#!/bin/bash
cd "$(dirname "$0")";
CWD="$(pwd)"
echo $CWD
python -c 'import proj_application.cron as cron; cron.test()'
And in cron.py,
from django.core.mail import send_mail
from smtplib import SMTP
from email.mime.text import MIMEText
import datetime
def test():
message = "<p>This is test mail scheduled to send every minute</p>"
my_email = MIMEText(message, "html")
my_email["From"] = "xxx#domain.com"
my_email["To"] = "yyy#domain.com"
my_email["Subject"] = "Title"
sender = "person1#domain.com"
receivers = ["person2#domain.com"]
with SMTP("localhost") as smtp:
smtp.login(sender, "yyy#1234")
smtp.sendmail(sender, receivers, my_email.as_string())
Actual Problem:
The crontab is now able to call the exec.sh file and I am able to print $CWD in echo, when the call comes to cron.py, It is unable to recognize django.core.mail and throws the following error.
from django.core.mail import send_mail
ImportError: No module named django.core.mail
I think, I need to set up virtual environment or python variable path somewhere, but since I am new to crontab, I am not sure how to do that.
Any assistance is appreciated. Thanks in advance.

The way I interpret this is that you are normally running your script in a virutal environment, and it is failing now that you added it to a cron job. If this is incorrect, just skip to the last paragraph.
To run a cron job through a virutal environment, you need to use the virtual environment's python as the python you want. E.g to run the cron.py file:
* * * * * /path/to/venv/bin/python3 /path/to/cron.py >> /Users/ashwin/dashboard/proj_application/data.log 2>&1
This is the way I would recommend doing it, as it doesn't seem like that shell script is entirely necessary (anything in there can easily be done at the top of the python script), but if it is you can do something similar where you call the virtual environment's python instead of the default python. like this:
/path/to/venv/bin/python3 -c 'import proj_application.cron as cron; cron.test()'
However, if this doesn't work, this may not be an issue with the cron job, and instead with the django setup of the mail client (e.g. making sure it is included in the installed apps, etc.)
Since it is a little unclear where your issue is (mail client setup, cron setup, venv configuration, etc), if you are sure it is an issue with running your scripts you usually run in a virtual environment, than these steps should help, otherwise I would make sure your mail client is configured correctly. Make sure your scripts are correct by running them in the console (not through a cron job), and then you can come back and update the post with more detailed info about where the problem you have lies.

yes you are correct you may need to use virtual environment(although its optional but best practice).
To create virtual environment(expects python to be installed prior)
python -m venv venv
To activate (path may differ based on OS(mine is windows))
source venv/Scripts/activate
To install dependency in virtual environment
pip install Django
Now you all set to use virtual env's Django.
which python should map to venv path.
Now you have two ways to run python script
#1 use direct python path
absolute/path/of/venv/bin/python3 -c 'import proj_application.cron as cron; cron.test()'
#2 activate virtual environment and use the bash script as same.
#!/bin/bash
cd "$(dirname "$0")";
CWD="$(pwd)"
echo $CWD
source venv/Scripts/python # this will be in windows
python -c 'import proj_application.cron as cron; cron.test()'

Error while uploading instructions using PyFirmata package

So I am trying to learn the PyFirmata module of python that allows us to write codes in Python and upload it to an Arduino Board.
I am learning it from this website.
Code that causes the inbuilt LED to blink:
import pyfirmata
import time
board = pyfirmata.Arduino('/dev/ttyACM0') #This Line gives error
while True:
board.digital[13].write(1)
time.sleep(1)
board.digital[13].write(0)
time.sleep(1)
But, When I run it( in Sublime Text 3) , I get this error:
raise SerialException("could not open port {!r}: {!r}".format(self.portstr, ctypes.WinError()))
serial.serialutil.SerialException: could not open port '/dev/ttyACM0': FileNotFoundError(2, 'The
system cannot find the path specified.', None, 3)
[Finished in 7.0s]
Can Anyone help me to fix this error?

The port you are trying to access doesn't exist or your program doesn't have permission to access it.
First checkout whether the port exist or not. If you have Arduino software installed, you can check from it. In case it's missing, another port should appear, smth like /dev/ttyACM*, where * is any number. Choose it and try running.
In case it exists, but the program still doesn't work visit the official arduino site.
This problem is solved in two command. Open terminal and type:
ls -l /dev/ttyACM*
Then you'll get something like this:
crw-rw---- 1 root dialout 188, 0 5 apr 23.01 ttyACM0
Here what we need is dialout. After the above command, type the below command, and the restart your device.
sudo usermod -a -G your_username
This should work, but there are cases, when it doesn't. Then you need to add your port to root access:
=============================================================================
cd ~/etc/udev/rules.d
In this directory, create a new .rules file and add the following to it:
KERNEL=="ttyACM0", MODE="0666"
This should work.

Calling Matlab scripts from Django with Python's Popen class

I'm developing a Django app which runs Matlab scripts with Python's Popen class. The python script that calls Matlab scripts lives in the main folder of my Django app (with views.py). When I call the script from command line, it runs like a charm but when I make a request from the client in order to run the corresponding python script, I receive the following warning:
"< M A T L A B (R) > Copyright 1984-2018 The MathWorks, Inc. R2018a (9.4.0.813654) 64-bit (glnxa64) February 23, 2018 To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com. >> [Warning: Unable to create preferences folder in /var/www/.matlab/R2018a. Preferences folder location must be writable. Using a temporary preferences folder for this MATLAB session. See the preferences documentation for more details.] >>
My app uses a Python virtual environment and it is being deployed with Apache web server.
Here is my python script that calls Matlab scripts:
import os
import subprocess as sp
import pymat_config
def pymat_run():
pwd = pymat_config.pwd_config['pwd']
cmd1 = "-r \"Arg_in = '/path/to/my/main/folder/input.txt'; Arg_out = '/path/to/my/main/folder/file.txt'; matlab_script1\""
baseCmd1 = ['/usr/local/MATLAB/R2018a/bin/matlab', '-nodesktop', '-nosplash', '-nodisplay', 'nojvm', cmd1]
os.chdir('/path/to/matlab_script1')
sudo_cmd = sp.Popen(['echo', pwd], stdout=sp.PIPE)
exec1 = sp.Popen(['sudo', '-S'] + baseCmd1, stdin=sudo_cmd.stdout, stdout=sp.PIPE, stderr=sp.PIPE)
out, err = exec1.communicate()
return out
Any suggestions ?

Finally I managed to find the solution of that issue by myself. The problem came from the kind of user who called the Matlab's script. When I was running the above script from a Python interpreter or from the shell, it was the user (with the user password) who was running the script while when I was calling the script from the client the user was the web server's user: www-data.
So at first to avoid the above warning I gave permissions to www-data user to the /var/www folder with the following command:
sudo chown -R www-data /var/www/
After that, the "Warning" disappeared but the script still didn't run because it was asking for www-data's password internally and taking user's password from pymat_config file.
To solve this, I edited /etc/sudoers file in order for www-data to be able to call Matlab scripts without asking password. So I added the following line:
www-data ALL=(ALL) NOPASSWD: /usr/local/MATLAB/R2018a/bin/matlab
and now it runs like a charm !

How to parse python module for openstack

Recently, when I was installing openstack on 3 vm on centos 7 using answer file I had the following error:
10.7.35.174_osclient.pp: [ ERROR ]
Applying Puppet manifests [ ERROR ]
ERROR : Error appeared during Puppet run: 10.7.35.174_osclient.pp
Error: Execution of '/usr/bin/yum -d 0 -e 0 -y list python-iso8601' returned 1: Error: No matching Packages to list
You will find full trace in log /var/tmp/packstack/20160318-124834-91QzZC/manifests/10.7.35.174_osclient.pp.log
Please check log file /var/tmp/packstack/20160318-124834-91QzZC/openstack-setup.log for more information
Additional information:
* Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
* File /root/keystonerc_admin has been created on OpenStack client host 10.7.35.174. To use the command line tools you need to source the file.
* To access the OpenStack Dashboard browse to http://10.7.35.174/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
I have already manually installed that module, but the same problem occures anyway.
That command only runs like that:
/usr/bin/yum -d 0 -e 0 -y list python2-iso8601
Is there any way to parse it to python?
Do you have any ideas how to solve it?

Found that kilo version works fine.

Connecting to a remote IPython instance

I would like to run an IPython instance on one machine and connect to it (over LAN) from a different process (to run some python commands). I understand that it is possible with zmq : http://ipython.org/ipython-doc/dev/development/ipythonzmq.html .
However, I can not find documentation on how to do it and whether it is even possible yet.
Any help would be appreciated!
EDIT
I would like to be able to connect to IPython kernel instance and send it python commands. However, this should not be done via a graphic tool (qtconsole) , but I want to be able to connect to that kernel instance from within a different python script...
e.g.
external.py
somehow_connect_to_ipython_kernel_instance
instance.run_command("a=6")

If you want to run code in a kernel from another Python program, the easiest way is to connect a BlockingKernelManager. The best example of this right now is Paul Ivanov's vim-ipython client, or IPython's own terminal client.
The gist:
ipython kernels write JSON connection files, in IPYTHONDIR/profile_<name>/security/kernel-<id>.json, which contain information necessary for various clients to connect and execute code.
KernelManagers are the objects that are used to communicate with kernels (execute code, receive results, etc.).
*
A working example:
In a shell, do ipython kernel (or ipython qtconsole, if you want to share a kernel with an already running GUI):
$> ipython kernel
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-6759.json
This wrote the 'kernel-6759.json' file
Then you can run this Python snippet to connect a KernelManager, and run some code:
from IPython.lib.kernel import find_connection_file
from IPython.zmq.blockingkernelmanager import BlockingKernelManager
# this is a helper method for turning a fraction of a connection-file name
# into a full path. If you already know the full path, you can just use that
cf = find_connection_file('6759')
km = BlockingKernelManager(connection_file=cf)
# load connection info and init communication
km.load_connection_file()
km.start_channels()
def run_cell(km, code):
# now we can run code. This is done on the shell channel
shell = km.shell_channel
print
print "running:"
print code
# execution is immediate and async, returning a UUID
msg_id = shell.execute(code)
# get_msg can block for a reply
reply = shell.get_msg()
status = reply['content']['status']
if status == 'ok':
print 'succeeded!'
elif status == 'error':
print 'failed!'
for line in reply['content']['traceback']:
print line
run_cell(km, 'a=5')
run_cell(km, 'b=0')
run_cell(km, 'c=a/b')
The output of a run:
running:
a=5
succeeded!
running:
b=0
succeeded!
running:
c=a/b
failed!
---------------------------------------------------------------------------
ZeroDivisionError Traceback (most recent call last)
/Users/minrk/<ipython-input-11-fb3f79bd285b> in <module>()
----> 1 c=a/b
ZeroDivisionError: integer division or modulo by zero
see the message spec for more information on how to interpret the reply. If relevant, stdout/err and display data will come over km.iopub_channel, and you can use the msg_id returned by shell.execute() to associate output with a given execution.
PS: I apologize for the quality of the documentation of these new features. We have a lot of writing to do.

If you just want to connect interactively, you can use SSH forwarding. I didn't find this documented anywhere on Stack Overflow yet, yet this question comes closest. This answer has been tested on Ipython 0.13. I got the information from this blog post.
Run ipython kernel on the remote machine:
user#remote:~$ ipython3 kernel
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing kernel-25333.json
Look at the kernel-25333.json file:
user#remote:~$ cat ~/.ipython/profile_default/security/kernel-25333.json
{
"stdin_port": 54985,
"ip": "127.0.0.1",
"hb_port": 50266,
"key": "da9c7ae2-02aa-47d4-8e67-e6153eb15366",
"shell_port": 50378,
"iopub_port": 49981
}
Set up port-forwarding on the local machine:
user#local:~$ ssh user#remote -f -N -L 54985:127.0.0.1:54985
user#local:~$ ssh user#remote -f -N -L 50266:127.0.0.1:50266
user#local:~$ ssh user#remote -f -N -L 50378:127.0.0.1:50378
user#local:~$ ssh user#remote -f -N -L 49981:127.0.0.1:49981
Copy the kernel-25333.json file to the local machine:
user#local:~$ rsync -av user#remote:.ipython/profile_default/security/kernel-25333.json ~/.ipython/profile_default/security/kernel-25333.json
Run ipython on the local machine using the new kernel:
user#local:~$ ipython3 console --existing kernel-25333.json
Python 3.2.3 (default, Oct 19 2012, 19:53:16)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.1.rc2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import socket; print(socket.gethostname())
remote

Update to minrk's answer after the split to jupyter. With
jupyter_client (4.1.1)
the simplest code is rather something like:
import jupyter_client
cf=jupyter_client.find_connection_file('6759')
km=jupyter_client.BlockingKernelClient(connection_file=cf)
km.load_connection_file()
km.execute('a=5')
Note that:
jupyter_client.BlockingKernelClient is also aliased with jupyter_client.client.BlockingKernelClient.
the shell (km.shell_channel) does not have the method execute() & get_msg() anymore.
Currently it is quite difficult to find an updated documentation; nothing yet on http://jupyter-client.readthedocs.org/en/latest/ for BlockingKernelClient. Some code in https://github.com/jupyter/jupyter_kernel_test. Any link welcome.

The above answers are a bit old. The solution for the latest version of ipython is much simpler but is not documented well at one place. So I thought I would document it here.
Solution to connect from any OS to a ipython kernel running on Windows
If either the client or server is a linux or other operating system, just change the location of kernel-1234.json appropriately based on Where is kernel-1234.json located in Jupyter under Windows?
On your windows based kernel start, make sure ipykernel is installed using pip install ipykernel
Start the ipykernel using ipython kernel -f kernel-1234.json
Locate the kernel-1234.json file on your Windows machine. The file will probably have a different number, not 1234 and will most likely be located in 'C:\Users\me\AppData\Roaming\jupyter\runtime\kernel-1234.json': https://stackoverflow.com/a/48332006/4752883
Install Jupyter Console (or Jupyter Qtconsole/notebook) using pip install jupyter-console or pip install qtconsole https://jupyter-console.readthedocs.io/en/latest/
If you are on Windows do a ipconfig to find out the ip address of your Windows server. (On Linux do a ifconfig at the shell prompt). In the kernel-1234.json file change the ip address from 127.0.0.1 to the ip address of your server. If you are connecting from another Windows server, then copy the kernel-1234.json file to your local computer and note down the path.
Navigate to the folder containing the kernel-1234.json and start Jupyter Console using jupyter console --existing kernel-1234.json

If you're using Anaconda, in OS X the JSON file is stored at
/Users/[username]/Library/Jupyter/runtime/
In Windows:
c:\Users[username]\AppData\Roaming\jupyter\runtime\