So I am trying to compare output of two python programs, which have files that I will call trace1.py and trace2.py. Currently I am using process substitution with diff to try and compare their outputs, however I'm having trouble with finding both files, since they are in separate sub-directories of my current directory:
diff <(python /subdir1/tracing1.py) <(python /subdir2/tracing2.py)
When I run this, I get
The system cannot find the file specified.
I think I'm messing up some sort of path formatting, or else I'm using the process substitution incorrectly.
EDIT: In the end I decided that I didn't need to use process substitution, and instead could just diff program output after each program is run. However thanks to Fallenreaper in the comments, I was able to find a single command that does what I initially wanted:
python subdir1/tracing1.py > outfile1.txt & python subdir2/tracing2.py > outfile2.txt & diff outfile1.txt outfile2.txt
Sorry, not enough rep to comment yet :(
Your line works perfectly when you remove that slash. I would suggest using absolute path names or a relative path from current directory cos that front slash would take you to your root directory.
Cheers.
Related
I've been tasked with finding out a way to search my entire hard drive for any given file using Python 2 in both Windows and Linux. I was excited to have pieced together various posts to concoct a solution, only to realize the paths being output by Python are not quite correct. Here are my results on Linux:
Unless the /home/pi directory is being queried 3 times from the for loops, I shouldn't be seeing the exact /home/pi/chromium-browser listed 3 times. I assume a couple of them are directories and one of them is the symlink/executable. However, the filepaths are [incorrectly] all the same.
Here is the same code on Windows:
Problem is, that is not where OS HW 2.docx is located. Rather, it's actually located in C:\Users\Wade\Dropbox\School\Fall 2018\IT344\HW2\OS HW 2.docx, as seen in the screenshot below:
There is not another instance of OS HW 2.docx in the root directory of C:\Users\Wade, as Python shell seems to indicate.
Where am I going wrong in my code? Is there a more accurate alternative to os.path.abspath()?
The files list in the tuples returned by os.walk contains only the file names without the path names, so you should join it with the path name before you call os.path.abspath:
print os.path.abspath(os.path.join(root, file))
I have files .sh files and .json files in which there are file paths given to point to a specific directory, but I should keep on changing the file path, depending on where my python scipt is run.
eg:content of one of my .sh file is
"cd /home/aswany/BotStudioInstallation/databricks/platform/databricksastro"
and I should change the file path via python code where the following path
"/home/aswany/BotStudioInstallation/" keep on changing depending on where databicks is located,
I tried the following code:
replaceAll(str(self.currentdirectory)+
"/databricks/platform/devsettings.json",
"/home/holmes/BotStudioInstallation",self.currentdirectory)
and function replaceAll is:
def replaceAll(self,file,searchExp,replaceExp):
for line in fileinput.input(file, inplace=1):
if searchExp in line:
line = line.replace(searchExp,replaceExp)
sys.stdout.write(line)
but above code only replaces a line
"home/holmes/BotStudioInstallation" to the current directory I am logged in,bt it cannot be sure that "home/holmes/BotStudioInstallation" is the only possibility it keep on changing like "home/aswany/BotStudioInstallation","home/dev3/BotStudioInstallation" etc ,I thought of regular expression for this.
please help me
Not sure I 100% understood your issue, but maybe I can help nonetheless.
As pointed out by J.F. Sebastian, you can use relative paths and remove the base part of the path. Using ./databricks/platform/devsettings.json might be enough. This is by far the most elegant solution.
If for any reason it is not, you can keep the directory you need to access, then append it to the base directory whenever you need it. That should allow you to deal with changes in the base directory. Though in the case the files will be used by other applications than your own, that might not be an option.
dir = get_dir_from_json()
dir_with_base = self.currentdirectory + dir
Alternatively, not an elegant solution though, without using regex you can use a "pattern" to always replace.
{
"directory": "<<_replace_me_>>/databricks/platform"
}
Then you know you can always replace "<<_replace_me_>>" with the base directory.
this is my first time posting to StackOverflow and I'm hoping someone can assist. I'm fairly new at pig scripts and have encountered a problem I can't solve.
Below is a pig script that fails when I attempt to write results to a file:
register 'myudf.py' using jython as myfuncs;
A = LOAD '$file_nm' USING PigStorage('$delimiter') AS ($fields);
B = FILTER A by ($field_nm) IS NOT NULL;
C = FOREACH B GENERATE ($field_nm) as fld;
D = GROUP C ALL;
E = FOREACH D GENERATE myfuncs.theResult(C.fld);
--DUMP E;
STORE E INTO 'myoutput/theResult';
EXEC;
I see the results of E when I Dump to the screen. However, I need to store the results temporarily in a file. After the Store command, the error I receive is: Output Location Validation Failed.
I've tried numerous workarounds, like removing the theResult folder and removing the earlier contents of theResult, but none of the commands I use work. These have been along the lines of:
hdfs dfs -rm myoutput/theResult
and
hadoop fs -rm myoutput/theResult
...using both the shell (hs) and file system (fs) commands. I've tried to call another function (shell script, python function, etc.) to clear the earlier results stored in the myoutput/theResult folder. I've read every website I can find and nothing is working. Any ideas??
the output location of a mapreduce is a directory. So, you must have tried it this way
hadoop fs -rmr myoutput/theResult
and then run the pig script. It will work.
"rmr" - remove recursive, which deletes both folder/file
"rm" - is just remove which removes only file
Everytime, you need to either change output path or delete and use the same, since HDFS is worm(write once read many) model storage.
Couple of things you can try-
making sure output director is a valid path.
Remove the entire directory and not just content within it. Remove directory with 'rmr and check that path doesn't exist before running pig script.
Thanks for both of your replies. I now have a solution that is working:
fs -mkdir -p myoutput/theResult
fs -rm -r myoutput/theResult
The first line attempts to create a directory, but the "-p" prevents an error if it already exists. Then the second line removes it. Either way, there will be a directory to remove, so no error!
The output of store is confusing when we are using Pig for the first time.
store grp into '/output1';
This will create the folder named 'output1' in root. The folder should not be already present
You can give your own hdfs path here like /user/thewhitetulip.
hdfs dfs -ls /output1
output:
/output1/_SUCCESS
/output1/part-r-00000
The part-r-00000 file is the output of the store program.
In Python3.4 (and also 2.6), using the os.path.relpath cmd, I get different answers if I run:
os.path.relpath("~/foo/bar.txt", "~")
than if I run
os.path.relpath("~/foo/bar.txt", "/home/<username>/")
(where I replace <username> with my user name, naturally!)
Namely, if I'm in my home directory, in the first case I get
foo/bar.txt
(as I would expect), but in the second case I get
~/foo/bar.txt
which is not what I would expect, given that ~ is exactly /home/<username>. (I get the same result in the second case regardless of whether I include the trailing '/' in the second arg.)
The second case gives an even more bizarre result if I run this within a subdirectory of my home directory, say ~/Data/:
Data/~/foo/bar.txt
which unless I'm mistaken is a non-existent directory.
I can figure out a work-around for this, but what is the logic for why it's happening this way? I don't suppose it's a bug, since it's been around a long time.
~ isn’t actually always your home directory—that’s only your shell’s interpretation. You can create a directory named ~ anywhere you want, just as you could create a directory named orange. Since ~ could be a plain old directory name, Python is treating it that way.
If you want Python to interpret ~ as meaning the home directory, as a shell would do, you need to tell Python you want to replace ~ with the path to the home directory using os.path.expanduser before passing your directory names to relpath.
While using the (excellent) Python autocompletion of YouCompleteMe (Jedi apparently) sometimes I'm suddenly unable to save because what vim thinks the path to my current file changes. When I open a file with vim bpython/curtsiesfrontend/repl.py, running
:echo expand('%:p')
gives me
/Users/tomb/Dropbox/code/bpython/bpython/curtsiesfrontend/repl.py
but after I use the completion, the same command gives
bpython/curtsiesfrontend/repl.py
and I can no longer save the file because that's not a path that exists. :pwd gives /Users/tomb/Dropbox/code/bpython before and after - so there must be some other part to the equation "what is the full directory path to this file."
I'm wondering what vim commands or concepts I should be looking at to identify the issue. There must be some concept of current directory of a file that is changing (however vim calculates the value of expand('%:p')) but I don't know what to call it. What is the concept of current directory that is changing?
The current file location is relative to Vim current path, which can be set with :cd or :lcd.
In order to identify where it is being changed you could use the 'verbose' option along the :redir command:
redir #a
set verbose=9
<execute the steps to reproduce the issue>
redir end
set verbose&
new
put a
Then search for 'chdir'. Increasing the value of 'verbose' displays more information, but it also make it harder to execute each step to reproduce the issue. After identifying a smaller test procedure you could repeat these steps using a higher value for 'verbose'.