Develop Reference

In Google Colab, specifying folder location as cd $path inside !Terminal with variable doesn't work. Why is this?

I have a confusing problem. I'm just trying to take a folder location from a colab form and combine all the mp4 videos in there into one video. However, I can't seem to provide this folder location via a variable. When hard coded, it works perfectly fine. What am I doing wrong here?
This code doesn't work because the cd command is not working somehow.
Location = "drive/MyDrive/MyFolder/" ##param {type:"string"}
!cd $Location; for f in *.mp4; do echo "file $f" >> videos.txt; done; sort --version-sort -o video.txt videos.txt; ffmpeg -f concat -i video.txt -c copy output.mp4; rm video.txt videos.txt
However when I hard code the location instead of $Location variable, it works without any issues. This code runs perfectly.
Location = "drive/MyDrive/MyFolder/" ##param {type:"string"}
!cd "drive/MyDrive/MyFolder/"; for f in *.mp4; do echo "file $f" >> videos.txt; done; sort --version-sort -o video.txt videos.txt; ffmpeg -f concat -i video.txt -c copy output.mp4; rm video.txt videos.txt
Can someone please tell me why earlier one doesn't work? It is so frustrating because every other shell command works fine when using $location. Only cd command seems not to work. Is this a colab specific issue?
P.S. Have to do in one line because all other commands happen inside that folder, so it's dependent on the folder being changed.
I tried using the variable as {Location} instead of $Location, but no difference. The suggested problem is irrelevant because I don't want to change the whole directory for the cell. I just want to go to that location and concat the videos, then get back to the usual directory.

env command not working with find command

Im trying to write a script:
env PYTHONPATH=$PYTHONPATH: $Dir/scripts find * -name ‘*.py' -exec pylint (} \\; | grep . && exit 1
The code is finding all scripts in the root directory instead of using the environmental variables I set. Any help on writing this code to only look for files in the directory I set as a value in PYTHONPATH.

env PYTHONPATH=$PYTHONPATH: $Dir/scripts isn't doing what you think it's doing. Including $PYTHONPATH includes the former value of PYTHONPATH, meaning whatever you have it already set to or a blank default. The space in your variable also makes it invalid, and instead interprets the $Dir/scripts as a new command. It looks like what you want would be env PYTHONPATH=$Dir/scripts — but there's actually an easier way.
If you have __init__.py files in your directory, you can just do pylint ./some-directory. If you don't, you can use xargs: find . -type f -name "*.py" | xargs pylint. If you wanted to pass the directory instead of have it coded to . (your current calling directory) you could do that too:
# set directory to first argument
dir="$1"
# check if "dir" was actually provided, if not, set to `.`
if [ -z "$dir" ]; then dir=.; fi
find "$dir" -type f -name "*.py" | xargs pylint
You could save that in a script or function and then call it either with a directory (like run-pylint-on-everything.sh ~/foo/bar, or not, in which case it would run starting from your current shell location.

There’s no space between the PYTHONPATH value, it was a typo mistake, I want to run the command on a CLI instead of a script.

Fastest way to merge a directory tree

I have multiple directories of the form
foo/bar/baz/alpha_1/beta/gamma/files/uniqueFile1
foo/bar/baz/alpha_2/beta/gamma/files/uniqueFile2
foo/bar/baz/alpha_3/beta/gamma/files/uniqueFile3
What is the fastest way to merge these directories to a single directory structure like
foo/bar/baz/alpha/beta/gamma/files/uniqueFile1...uniqueFile3
I could write a python script to do that but is there a faster way to do that on a debian machine ? Can rsync help in this case ?
EDIT:
Apologies for not making it clear earlier, the depth in the examples is ~10-12 and I do not know the some directory names such as alpha*, these are randomly generated while throwing out logs. I was using find with wildcards to list these files earlier but now another level has been added in the path, that caused my find queries to take over a minute from 0.004s. So I am looking for a faster solution.
/known_fixed_path_5_levels/*/known_name*/*/fixed_path_2_levels/n_unique_files
has become
/known_fixed_path_5_levels/*/known_name*/*/xx*/fixed_path_2_levels/unique_file_1
/known_fixed_path_5_levels/*/known_name*/*/xx*/fixed_path_2_levels/unique_file_2
.
.
/known_fixed_path_5_levels/*/known_name*/*/xx*/fixed_path_2_levels/unique_file_n
I basically want to collect all those unique files into one place like how it was before.

With find:
mkdir --parents foo/bar/baz/alpha/beta/gamma/files; #create target directory if nessessary
find foo/bar/baz/alpha_[1-3]/beta/gamma/files -type f -exec cp {} foo/bar/baz/alpha/beta/gamma/files \;

As question is not clear about copying or moving, there is two ways, without copy! Even second part don't effectively copy your data!
Simple bash command
Simply:
cd foo/bar/baz
mv -it alpha/beta/gamma/files alpha_*/beta/gamma/files/uniqueFile*
with -i switch to prevent overwritting.
This will work perfectly for small bunch of files.
More robust and adaptive find syntax
Or by using find:
cd foo/bar/baz
find alpha_* -type f -mindepth 3 -exec mv -it alpha/beta/gamma/files {} +
Advantage of using find are
you could add a lot of flags like -name, -mtime and so on
find will never try to pass more files to command (mv) that command line could hold.
cp -al specific UN*X concept
Under Un*x, you could create hard-link wich is not symbolic links, but a secondary entry in folder tree, for the same inode.
Nota: As only one inode has to be referenced, this could work only on same filesystem.
By using
cp -ialt alpha/beta/gamma/files alpha_*/beta/gamma/files/uniqueFile*
You will copy in one directory all inodes references, but keeping only one file for each.
Using bash's globstar feature:
cd foo/bar/baz
shopt -s globstar
cp -alit alpha/beta/gamma/files alpha_*/**/uniqueFile*

How to create a working directory using mkdir command in Mac OS X?

I started learning python recently and I am very new to programming.
My book tells the following :
On Unix-based systems (including Mac OS X and Linux), your working directory might be in /usr/home and be created by a mkdir command in a shell window or file explorer GUI specific to your platform, but the same concepts apply. The Cyg- win Unix-like system for Windows is similar too, though your directory names may vary (/home and /cygdrive/c are candidates).
I am running python on a Mac OS X and I am finding hard to create a directory.
What code should I type in the Terminal ?
I tried :
mkdir
It says :usage: mkdir [-pv] [-m mode] directory ...
cd c: mkdir
It outputs : -bash: cd: c:: No such file or directory
Please help me
Thanks

mkdir mydir creates a directory called mydir; then you could change directory like cd mydir. To get further information you could check out the man page for mkdir using man mkdir from your terminal. I'm not really sure what you mean with working directory. The pwd command prints out the current working directory, i. e. the directory you are currently standing in. – Cyclone

For making directory in mac use below command:
mkdir nameOfDir
If you want to check directory then use below command:
cd nameOfDir

The problem is with the name you want to use for your directory. If the name contains any characters as '#', '$' etc., then it might be considered as another command in the bash. Try to name your directory using only lowercase alphabets and underscores because of its convention(at least as far as I know).
If you want to still name it using some unconventional characters then use '' to tell the machine that it should not be considered as a command. It might look something like this.
$ mkdir \#659div2
Here character '#' is the unconventional character. Or like this
$ mkdir images\ from\ tour
Here character ' '(space) is an unconventional character.

ERROR virtualenvwrapper in GitBash

I trying to setup virtualenvwrapper in GitBash (Windows 7). I write the next lines:
1 $ export WORKON_HOME=$HOME/.virtualenvs
2 $ export MSYS_HOME=/c/msys/1.0
3 $ source /usr/local/bin/virtualenvwrapper.sh
And the last line give me an error:
source /usr/local/bin/virtualenvwrapper.sh
sh.exe: /usr/local/bin/virtualenvwrapper.sh: No such file or directory
I find, where on my drive is virtualenvwrapper.sh and change directory name. On my computer it's /c/Python27/Scripts/virtualenvwrapper.sh. When I again run command
$source /c/Python27/Scripts/virtualenvwrapper.sh
I get the next ERROR message:
sh.exe":mktemp:command not found ERROR: virtualenvwrapper could not create a temporary file name
I check my enviroment variable: C:\python27\;C:\python27\scripts\;C:\python27\scripts\virtualenvwrapper.sh\;C:\msys;C:\Program Files (x86)\Git\cmd;C:\Program Files (x86)\Git\bin\
I don't know where i made a mistake

The error is saying that sh.exe (the shell) can't find a command matching mktemp, which means it's not present in GitBash, at least not in your environment.
One option is you could download a Windows version of mktemp, such as http://gnuwin32.sourceforge.net/packages/mktemp.htm and then place it in the C:\Program Files (x86)\Git\bin directory. The shell should then be able to match the mktemp command and be able to proceed.

I've found a fix for this problem on a Windows 8 machine using GitBash.
TL;DR:
Get mktemp for windows, put it somewhere that can be used by GitBash, then edit virtualenvwrapper.sh and on line 202, add a touch command with the file created. It should look like this:
file="$(virtualenvwrapper_mktemp -t virtualenvwrapper-$suffix-XXXXXXXXXX)"
touch $file # this is the new line
if [ $? -ne 0 ] || [ -z "$file" ] || [ ! -f "$file" ]
FULL ANSWER:
As khampson mentioned, you do have to download mktemp and place it where your Git\bin (C:\Program Files (x86)\Git\bin usually) directory is. After that, running the virtualenvwrapper.sh file would cause an error saying:
path = C:/Users/User/AppData/Local/Temp/virtualenvwrapper-initialize-hook-XXXXXX XXXX
lpPathBuffer = C:\Users\User\AppData\Local\Temp\
szTempName = C:\Users\User\AppData\Local\Temp\tmp23A9.tmp
path = C:\Users\User\AppData\Local\Temp\tmp23A9.tmp
fd = 3
ERROR: virtualenvwrapper could not create a temporary file name.
On line 202(source), you see a function call to virtualenvwrapper_mktemp (which is just a wrapper function to call mktemp) and this is supposed to create the new temp file, but apparently it doesn't on windows.
Going through the manual for mktemp, on the examples section, you see that they are always sending something to that new file handle which forces the file to be created.
So instead of sending an empty string using echo like the manual, just add a touch command to the virtualenvwrapper.sh:
file="$(virtualenvwrapper_mktemp -t virtualenvwrapper-$suffix-XXXXXXXXXX)"
touch $file # new command here
This should force windows to create the temp file. I can't post the rest of the links due to low rep but I hope this still helps someone.
EDIT
I created a pull request on the virtualenvwrapper repo and it got approved. You can see the touch command I suggested added here.