I have several different Notebooks in Google Colab that I use to automate some tasks. I want to create a single Notebook that runs all this tasks, so I can open a single tab with one single Notebook, run it and it will run all other tasks inside these different Notebooks.
I have two questions regarding this problem:
The solution I found is not working (I will describe it below). How
do I make it work?
Is there a better solution than the one I found?
About the first question:
Image I have Notebook_1 and Notebook_2 each one with a bunch of functions that automate my tasks. What I am doing is, downloading them as Notebook_1.py and Notebook_2.py, saving these files in a Google Drive folder. Then in my Notebook_main, which is the notebook that should house all notebooks, I run:
# Mounts Google Drive
from google.colab import drive
drive.mount('/content/drive')
# Copies .py files into Google Colab
!cp /content/drive/MyDrive/my_modules/Notebook_1.py /content
!cp /content/drive/MyDrive/my_modules/Notebook_2.py /content
# Import Modules
import Notebook_1
import Notebook_2
If I want to run a simple function inside these modules I just do:
Notebook_1.run_simple_function()
and this works. My problem happens when the function I am trying to run from the Notebook_1, for example, uses another module. Then I get the following error:
name 'os' is not defined
I imagine it happens because inside Notebook_1.py I call:
import os
...
os.remove(os.path.join(dir_download, item))
...
And I also think this will happen with all the modules that I call inside Notebook_1.py.
I have tried importing theses modules in Notebook_main, but it did not work. I do not know how to fix this. I need help.
Another issue is that I use a lot of Selenium, which needs to be installed in Google Colab before being imported. So I need to install and import it in Notebook_main and when I run a function with Notebook_1.run_function_that_uses_selenium() it should use the import from Notebook_main.
The second question is simpler. I just want to know if there is a better way to achieve the same result, i.e. run different Notebooks in Google Colab from a single notebook.
My constriction is that I can only use Google Colab and other Google related Platforms, I can not run anything locally.
Related
I am using Python in a Jupyter Lab notebook in a Docker container. I have the following code in one cell:
import numpy as np
import os
import pandas as pd
Then I run the following cell:
!pipreqs /app/loaded_reqs
and get:
INFO: Successfully saved requirements file in /app/loaded_reqs/requirements.txt
But when I open the requirements.txt, it shows up empty/blank. I expected numpy, os and pandas to be in this requirements.txt file. Why might it not be working?
According to this Medium post by Iván Lengyel, pipreqs doesn't support Jupyter notebooks. (This issue in in the pipreqs repo, open since 2016 convinces me of the veracity of that assertion. Nicely, the issue post also suggests the solution I had already found when searching the terms 'pipreqs jupyter' at Google.) Plus, importantly you generally don't use tools that act on notebook files inside the notebook you are trying to use. (Or at least it is something to always watch out for, [or test if possible], similar in a way to avoiding iterating on a list you are modifying in the loop.)
Solution -- use pipreqsnb instead:
In that Medium post saying it doesn't work with notebooks, Iván Lengyel proffers a wrapper for it that works for notebooks. So in the terminal outside the notebook, but in the same environment (inside the docker container, in your case), install pipreqsnb via pip install pipreqsnb. Then run it pointing it at your specific notebook file. I'll give an example in the next paragraph.
I just tried it and it worked in temporary sessions launched from here by pressing launch binder badge there. When the session came up, I opened a terminal and ran pip install pipreqsnb and then pipreqsnb index.ipynb. That first time I saw requirements.txt get made with details on the versions of matplotlib, numpy, scipy, and seaborn. To fully test it was working, I opened index.ipynb in the running session and added a cell with import pandas as pd typed in it and saved the notebook. Then I shutdown the kernel and over in the terminal ran, pipreqsnb index.ipynb. When I re-examined the requirements.txt file now pandas has been added with details about the versions.
More about maybe why !pipreqs /app/loaded_reqs failed:
I had the idea that maybe you needed to save the notebook first after adding the import statements cell? However, nevermind. That still won't help because as stated here pipreqs, and further confirmed at the pipreqs issues list doesn't support Jupyter notebooks.
Also, keep in mind the use of the exclamation in a notebook to run a command in the shell doesn't mean that shell will be in the same environment as the kernel of the notebook, see the second paragraph here to more perspective on that. (This can be useful to understand for future things though, such as why you want to use the %pip or %conda magic commands when installing from inside a notebook, see here, and not put an exclamation point in front of that command in modern Jupyter.)
Or inside the notebook at the end, I'd suggest trying %watermark --iversions, see watermark. And then making some code to generate the requirements.txt from that. (Also, I had seen there was bug in that related to some packages imported with from X import Y, see here.)
Or I'd suggest trying %pip freeze inside the notebook for the full environment information. Not just what the file needs, though.
I'm working with google-colab.
Can it be possible to call a function from one colab file in to another colab project, as we do in python like importing file.
ex.
I have two colab files 1.ipynb and 2.ipynb.
Let's say 1.ipynb has ABC function.
Now I want to utilize this already implemented ABC function into 2.ipynb
is there any way I can do it? or it's not possible?
It's possible using the import-ipynb libary:
!pip install import-ipynb
import import_ipynb
This lets you import code from .ipynb files in the same way you would from .py files - simply by calling import filename or from filename import function, class, etc. (which is more advisable).
Assuming you have your notebooks neatly organized on your Google Drive, eg. in MyDrive/Colab_Notebooks/ you can then mount it:
from google.colab import drive
drive.mount('/content/drive')
and import the functions you want from the source notebook:
from drive.MyDrive.Colab_Notebooks.notebook1 import ABC, someOtherFunction
A very simple question with a lot of context behind it, so thanks in advance for your patience. In summary I am attempting to control a DAC/ADC development board by adapting already existing software written in python. However, I am encountering an error when I simply try to import some of the prerequisites.
The board I am attempting to control is the ZCU111. The ZCU111 has an ARM processor on board and some wonderful folks found a way to put linux on board; Pynq. The pynq software is imaged onto a SD card, and the SD card is mounted onto the ZCU111 board and the processor boots from the card. I communicate to the board through a USB serial interface and a hosted SSH server on the ZCU111.
Using pynq, some modules have been created to control automatically set the inner workings of the ZCU111 board, ZCU111 GitHub.
Using those modules someone created some great Jupiter notebooks that help run the ZCU111 though a nice gui interface; RFSoC_SAM. However, I dont want run the board through a notebook or gui, I wanted to adapt parts of the code to a much simpler .py file to be run from the terminal.
image of filing system on SD card
To the left is an image of the filing system on the SD card. The GitHub folder contains the modules to control the inner workings, and its identical to the link above. The Jupiter notebook folder contains the notebook which I wish to emulate and works as expected. The pynq folder contains the modules for pynq itself. The Sami_py folder is where I placed my test code.
Here are where my problems and questions begin: The Notebook that works begins with 2 lines of code:
from rfsoc_sam.overlay import
sam = Overlay()
When I scour the SD card, I can't find rfsoc_sam anywhere on the SD card. I'm confused how it works? The GitHub listed above for the RFSoC_Sam DOES have the accompanying .py files. Why do'nt those .py files appear on my SC card? Does the Jupiter notebook package all the necessary files? Regardless the first step in creating my own software is to import the same modules the Overlay from the rfsoc_sam module does. Despite it not appearing on the SD card, I can open the file from the GitHub
from pynq import Overlay, allocate
import xrfclk
import xrfdc
import os
from .hierarchies import *
from .quick_widgets import Image
from ipywidgets import IntProgress
from IPython.display import display
from IPython.display import clear_output
import time
import threading
That is everything that needed to be imported, however I just wanted to start with xrfclk. Since that folder and init file can be found on the SD card in the GitHub folder, the same folder from the link above, ZCU111 GitHub.
I wrote a .py file placed in the Sami_py folder:
import sys
sys.path.insert(1, '//192.168.2.99/xilinx/GitHub/ZCU111-PYNQ/ZCU111/packages/xrfclk')
import xrfclk
The error message I receive:
Traceback (most recent call last):
File "Sami_py/RFSoC_Trial.py", line 3, in <module>
import xrfclk
ImportError: No module named xrfclk
I thought I pointed to the right directory, do i need to point to the directory that has the init file directly? I am not sure why i can't get the include to work, any thoughts?
Happy to provide more context. Thanks in advance for any help or advnce,
Sami
I'm sorry to hear you're having problems with RFSoC SAM. I created this project and can give you some help about RFSoC and PYNQ.
Firstly, the RFSoC SAM package is installed and maintained using pip, which is the python package manager. As PYNQ v2.6 uses Python3, we need to invoke the pip3 command to install RFSoC SAM. In the repository readme, you will see that installing RFSoC SAM on the ZCU111 has a few steps. You need to connect your board to the internet for these steps to work correctly. You should also use the terminal in the Jupyter Labs environment to run the commands (this is detailed in the repository readme). The first step is a patch for the xrfdc package, and looks a bit like this:
mkdir /home/xilinx/GitHub cd /home/xilinx/GitHub/ git clone https://github.com/dnorthcote/ZCU111-PYNQ cd /home/xilinx/GitHub/ZCU111-PYNQ cp /home/xilinx/GitHub/ZCU111-PYNQ/ZCU111/packages/xrfdc/pkg/xrfdc/__init__.py /usr/local/lib/python3.6/dist-packages/xrfdc/__init__.py
The reason a patch is required is because the xrfdc package needs an update that has not yet been implemented for the ZCU111, but it has been implemented for the RFSoC2x2. I simply moved these updates over to the ZCU111, which means RFSoC SAM will function correctly. If you do not implement this patch, the notebook for RFSoC SAM will fail to load. I am not in charge of the xrfdc package, which is why the package needs to be patched (not version controlled and updated).
After implementing the xrfdc patch, you can install RFSoC SAM via pip. The command looks a bit like this:
pip3 install git+https://github.com/strath-sdr/rfsoc_sam
This command will install the RFSoC SAM package to the following location:
/usr/local/lib/python3.6/dist-packages/rfsoc-sam
This location is where all pip packages are stored for PYNQ v2.6. You should not attempt to directly modify packages in this location as they are all under version control. If you would like to develop and change the RFSoC SAM source code, then you need to consider a slightly different workflow.
The RFSoC SAM package does not have any developer tools. To make changes to the project, I simply Git clone the project to my Jupyter workspace in PYNQ, and then make changes to the project. When I'm ready, I can simply reinstall the package by running the following command in a Jupyter Labs terminal:
pip3 install .
This command will just reinstall RFSoC SAM, but using the offline changes you make instead. Using this workflow, you could easily create a fork of RFSoC SAM, make changes to the source code, commit those changes, and easily install the package on your ZCU111. This workflow is something you should consider.
You may be more interested in trying out RFSoC + PYNQ projects that are of lower complexity. For instance, you will be able to see the following projects: RFSoC QPSK and RFSoC Radio. You may have also seen the RFSoC Workshop.
Going forward, I would recommend investigating the above projects, and also understanding Linux and pip a little better. A few google searches should get you on the right path here. Try to create your own pip package and a Github repository to store the package. If you can do this, you will have greatly improved your understanding of pip, and the PYNQ operating system.
Also, If you have questions in the future, you can use the [PYNQ discussion page] (https://discuss.pynq.io/). I will be able to find your questions faster here.
If you have any further issues, feel free to drop me a message. You can also use the RFSoC SAM issues page in the repository, which helps me track these type of questions a little easier.
David N.
I forked a repo from github and copied to my local machine and opened all the files in Jupyter. Now I have access to all the .py files and all the notebooks.
Let's say I want to add a new function to the package as follows:
teste(self):
return self
I do this by writing the function in the here.py file and to make sure it works I test it on a notebook by calling it in a cell and executing that cell:
print(here.teste(worked))
However, this doesn't work. My guess is that I have not updated the package itself so the function teste() does not exist. How do I commit this change to the package locally (without using pull request).
Most likely you need to restart your jupyter kernel for the changes to take effect.
Git is merely a versioning system, it does not care what python does and does not influence how python works.
Python loads your package when it is imported import my_package as mp. When you make changes to that package while python is running, it is not aware of those changes. If you try to re-import, python will merely check if it is already imported (which is true) and do nothing. So the changes still does not take effect. Only when you restart the kernel, and import the package will it take effect. You can also re-import a package with the following (python 3.4 and greater):
import importlib
importlib.reload(package)
I have set up a jupyter notebook server for multiple users to run notebooks. I want to provide modules that can fetch data and do some pre-processing. Since the data and data processing code is proprietary, I don't want the users to have access to the source code which can be done via import inspect module.
I have two questions:
Is there a way to prevent inspect module from loading? I have seen this in Quantopian notebooks where importing inspect module throws an error.
Are there other ways to prevent access to the source code of the modules?