I am currently trying to make python tutorials and host them using an ipython notebook on a Heroku site. The problem is that ipython notebooks are static when uploaded. I am trying to make it such that the user can use the notebook interactively (such as print outputs). I also dont want the output from their notebooks to be saved permanently on the Heroku website.
From what I understand, you have 2 issues do deal with :
interactive notebooks
"read only" notebooks (do not save the modifications)
For issue 1, you need to use a jupyter (the new IPython name for notebooks) server. Only showing the notebook is not enough because you need a server to "understand" and execute the modifications. See : http://jupyter-notebook.readthedocs.io/en/latest/public_server.html
I am not familiar with Heroku, after googling 2s I found this : https://github.com/pl31/heroku-jupyter which was able to deploy a working Jupyter server on a demo heroku machine.
According to me, issue 2 is more difficult to solve.
When the "learners" will change the notebook, the modifications will be applied to the notebook file (.ipnb) so the modifications will be persistent... This is not want you want.
You could try some tricks using file permissions to prevent the kernel to save the file, but I think it would only crash the kernel...
Moreover it asks several user-interaction problems, for instance what if I lose my internet connection ? Will I loose my work ? Why ? Is this what I really want as a learner ?
For this, the best solution is to provide a user access to the notebook / a worksapce where she can save her progression, but it is more work than just deploy a jupyter server. As an example, see databricks.com (the first (only) one that come to mind, not necessary the best).
(As a remark, it seems that the multi user mode is already implemented : https://jupyterhub.readthedocs.io/en/latest/)
I would like to add a last remark about the security of the server. Letting stranger access a server with an embedded shell sound like a bad idea if you are not prepared for the consequences. I would suggest you to see how you can put each user's jupyter session in a "jail" / container, anything that works in Heroku.
Related
I would like to create a self-contained, .exe file that launches a JupyterLab server as an IDE on a physical server which doesn't have Python installed itself.
The idea is to deploy it as part of an ETL workflow tool, so that it can be used to view notebooks that will contain the ETL steps in a relatively easily digestible format (the notebooks will be used as pipelines via papermill and scrapbook - not really relevant here).
While I can use Pyinstaller to bundle JupyterLab as a package, there isn't a way to launch it on the Pythonless server (that I can see), and I can't figure out a way to do it using Python code alone.
Is it possible to package JupyterLab this way so that I can run the .exe on the server and then connect to 127.0.0.1:8888 on the server to view a notebook?
I have tried using the link below as a starting point, but I think I'm missing something as no server seems to start using this code alone, and I'm not sure how I would execute this via a tornado server etc.:
https://gist.github.com/bollwyvl/bd56b58ba0a078534272043327c52bd1
I would really appreciate any ideas, help, or somebody to tell my why this idea is impossible madness!
Thanks!
Phil.
P.S. I should add that Docker isn't an option here :( I've done this before using Docker and it's extremely easy.
I am using a locked down system where I cannot install any applications including Anaconda or any other python.
Anybody knows if it is possible to access local files from a jupyter online solution? I know it would probably be slow as the file would have to be moved back and forth?
Thanks
Yes, you can use your local files from a Jupyter online solution doing what you say of moving them back and forth. (The remote server cannot connect to your local system itself beyond the browser sandbox, and so concerns like Chris mentions aren't an issue.)
I can demonstrate this easily:
Go to here and click on the launch binder badge you see.
A temporary session backed by MyBinder.org will spin up. Depending where you are in the world, you may be on a machine run by Jupyter folks via Google or another member of this service backed by folks in a Federation that believe this a valuable service to offer to empower Jupyter users.
After the session comes up, you'll be in JupyterLab interface. You'll see a file/directory navigation pane on the left side. You can click and drag a file on your local computer and drop it in that pane. You should see it show up on the remote directory.
You should be able to open and edit it. Depending on what it is or what you convert it to. You can even run it.
Of course you can make a new notebook on the remote session and save it. Then after saving it, download it back to your local machine by right-clicking on the icon for it in the file navigation pane and selecting 'Download'.
If you prefer to work in the classic Jupyter notebook interface, you can go to 'Help' and select 'Launch Classic Notebook' from the menu. The classic Jupyter Dashboard will come up. You will need to upload things to there using the upload button as drag and drop only works in JupyterLab. You can download back to your local computer from the dashboard or when you have a notebook open, you can use the file menu to download back to your local machine, too.
Make sure you save anything you make that is useful back to your machine as the sessions are temporary and will time out after 10 minutes of inactivity. They'll also disconnect after a few hours even if you are actively using them. There's a safety net built in that works if it does disconnect but you have to aware of it ahead of time. And it is best tested a few times in advance when you don't need it. See Getting your notebook after your Binder has stopped.
As this is going to a remote machine, obviously there are security concerns. Part of this is addressed by the temporary nature of the sessions. Nothing is stored remotely once the active session goes away. (Hence, the paragraph above because once it is gone, it is gone.) However, don't upload anything you wouldn't want someone else to see. Don't share keys and things with this system. In fact, it is possible now to do real time co-authoring/co-editing of Jupyter notebooks via the MyBinder system although some of the minor glitches are still being worked out.
A lot of packages you can install right in the session using %pip install or %conda install in cells right in the notebook. However, sometimes you want them already installed so the session is ready with the necessary software. (Plus some software won't work unless installed during the building backing image of the container backing the session.) That is where it becomes handy that you can customize the session that comes up via configuration files in public repositories. A list of places you can host those files is seen by going to MyBinder.org and pressing the dropdown menu in the top left side of that form there, under 'GitHub repository name or URL'. Here's an example. You can look in requirements.txt and see I install quite a few packages in the data science stack.
Of course there's other related online offerings for Jupyter (or you can install it on remote servers) and many use authentication. As some of those cost money and you are unsure about your locked system, the MyBinder.org system may help you test the limits of what you can do on your machine.
I am trying to deploy Django based website on AWS EC2. I have successfully created an instance and install my python libraries there. I am using Postgres here. For that, I have installed Postgres along with pgAdmin, but for some reason, it does not open. It just displayed that it's starting up the server but it does not open at all.
I am new to it so I do not know much about it. Can someone please help or guide me why it does not open up?
You will need to check the logs from
C:\Users\Administrator\AppData\Local with name - 'pgadmin4.startup'
A lot of the time removing the instance and recreating usually works but without seeing the logs it's hard to tell what the issue might be. Could also be worth making the instance a bit beefier as pgadmin does use a good amount of CPU and memory.
I created a jupyter notebook with the purpose of taking a survey on a fairly large group of people, which consists of 1 script that each person has to run and fill in. To make it convenient for them I hosted a public jupyter notebook server and mailed every person the link to participate.
The problem is that when one person is running the script, all other people have to wait until that person closes the notebook in order for them to run it. I want a system that generates one seperate kernel for every incoming connection so multiple people can take the survey at the same time.
Does anyone have any ideas?
Jupyter Notebook wasn't made for simultaneous collaboration on the same file. One solution I've seen that addresses exactly this problem is Google Colab, which is a fork of Jupyter built on Google's collaborative Docs platform, and allows exactly what you're talking about.
It looks like for Jupyter Lab, they're hoping to integrate simultaneous editing as a core feature (they were originally going for a Google Drive backend, but Google seems to have pulled support and now they're considering more P2P solutions like IPFS), but it looks like that work has hit a few roadbumps and won't be released with version 1.0.
As part of an effort to make the scikit-image examples gallery interactive, I would like to build a web service that receives a Python code snippet, executes it, and provides me with the generated output image.
For safety, the Python instances launched should be sandboxed and resource controlled, so I was thinking of using LXC containers.
Is this a good way to approach the problem? If so, what is the recommended way of launching one Python VM per request?
Stefan, perhaps "Docker" could be of use? I get the impression that you could constrain the VM that the application is run in -- an example web service:
http://docs.docker.io/en/latest/examples/python_web_app/
You could try running the application on Digital Ocean, like so:
https://www.digitalocean.com/community/articles/how-to-install-and-use-docker-getting-started
[disclaimer: I'm an engineer at Continuum working on Wakari]
Wakari Enterprise (http://enterprise.wakari.io) is aiming to do exactly this, and we're hoping to back-port the functionality into Wakari Cloud (http://wakari.io) so "published" IPython Notebooks can have some knobs on them for variable input control, then they can be "invoked" in a sandboxed state, and then the output given back to the user.
However for things that exist now, you should look at Sage Notebook. A few years ago several people worked hard on a Sage Notebook Cell Server that could do exactly what you were asking for: execute small code snippets. I haven't followed it since then, but it seems it is still alive and well from a quick search:
http://sagecell.sagemath.org/?q=ejwwif
http://sagecell.sagemath.org
http://www.sagemath.org/eval.html
For the last URL, check out Graphics->Mandelbrot and you can see that Sage already has some great capabilities for UI widgets that are tied to the "cell execution".
I think docker is the way to go for this. The instances are very light weight, and docker is designed to spawn 100s of instances at a time (Spin up time is fractions of a second vs traditional VMs couple of seconds). Configured correctly I believe it also gives you a complete sandboxed environment. Then it matters not about trying to sandbox python :-D
I'm not sure if you really have to go as far as setting up LXC containers:
There is seccomp-nurse, a Python sandbox that leverages the seccomp feature of the Linux kernel.
Another option would be to use PyPy, which has explicit support for sandboxing out of the box.
In any case, do not use pysandbox, it is broken by design and has severe security risks.