Develop Reference

How do I input strings in Linux terminal that points to file path using subprocess.call command?

I'm using Ubuntu and have been trying to do a simple automation that requires me to input the [name of website] and the [file path] onto a list of command lines. I'm using subprocess and call function. I tried something simpler first using the "ls" command.
from subprocess import call
text = raw_input("> ")
("ls", "%s") % (text)
These returned as "buffsize must be an integer". I tried to found out what it was and apparently I had to pass the command as a list. So I tried doing it on the main thing im trying to code.
from subprocess import call
file_path = raw_input("> ")
site_name = raw_input("> ")
call("thug", -FZM -W "%s" -n "%s") % (site_name, file_path)
These passed as an invalid syntax on the first "%s". Can anyone point me to the correct direction?

You cannot use % on a tuple.
("ls", "%s") % text # Broken
You probably mean
("ls", "%s" % text)
But just "%s" % string is obviously going to return simply string, so there is no need to use formatting here.
("ls", text)
This still does nothing useful; did you forget the call?
You also cannot have unquoted strings in the argument to call.
call("thug", -FZM -W "%s" -n "%s") % (site_name, file_path) # broken
needs to have -FZM and -W quoted, and again, if you use format strings, the formatting needs to happen adjacent to the format string.
call(["thug", "-FZM", "-W", site_name, "-n", file_path])
Notice also how the first argument to call() is either a proper list, or a long single string (in which case you need shell=True, which you want to avoid if you can).
If you are writing new scripts, you most definitely should be thinking seriously about targetting Python 3 (in which case you want to pivot to subprocess.run() and input() instead of raw_input() too). Python 2 is already past its originally announced end-of-life date, though it was pushed back a few years because Py3k adoption was still slow a few years ago. It no longer is, and shouldn't be -- you want to be in Py3, that's where the future is.

Here is a complete example of how you would call a executable python file with subprocess.call Using argparse to properly parse the input.
Your python file to be called (sandboxArgParse.py):
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--filePath", help="Just A test", dest='filePath')
parser.add_argument("--siteName", help="Just A test", dest='siteName')
args = parser.parse_args()
print args.siteName
print args.filePath
Your calling python file:
from subprocess import call
call(["python","/users/dev/python/sandboxArgParse.py", "--filePath", "abcd.txt", "--siteName", "www.google.com"])

Pyspark error handling with file name having spaces

I am using pyspark 2.1
Problem Statement: Need to validate the hdfs path, file if exist need to copy file name into variable
Below is the code used so far after referring few websites and stackoverflow
import os
import subprocess
import pandas as pd
import times
def run_cmd(args_list):
print('Running system command: {0}'.format(' '.join(args_list)))
proc = subprocess.Popen(args_list, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
proc.communicate()
return proc.returncode
today = datetime.now().date().strftime('%d%b%Y')
source_dir = '/user/dev/input/'+ today
hdfs_file_path=source_dir+'\'student marks details.csv\''
cmd = ['hdfs', 'dfs', '-find','{}','-name', hdfs_file_path]
code=run_cmd(cmd)
if code<>1:
print 'file doesnot exist'
System.exit(1)
else:
print 'file exist'
With above code I am getting error as "File doesn't exist" but file is present in that folder
Problem is able to run the run below command in shell console I am getting the complete path.
hdfs dfs -find () -name /user/dev/input/08Aug2017/'student marks details.csv'
When I tried to import in pyspark with above detailed code I am not able to execute as there exist space in an filename . Please help me in resolving this issue.

The problem
Your problem is on this line:
hdfs_file_path = source_dir + '\'student marks details.csv\''
You are adding two unneeded single quotes, and also forgetting to add a directory separator.
The reason the path works in this command:
hdfs dfs -find () -name /user/dev/input/08Aug2017/'student marks details.csv'
is because this is a shell command. On the shell that you are using (presumably it is bash), the following commands are equivalent:
echo '/user/dev/input/08Aug2017/student marks details.csv'
echo /user/dev/input/08Aug2017/'student marks details.csv'
bash removes the quotes, and merges the strings together, yielding the same string result, which is /user/dev/input/08Aug2017/student marks details.csv. The quotes are not actually part of the path, but just a way to tell bash to not split the string at the spaces, but create a single string, and then remove the quotes.
When you write:
hdfs_file_path = source_dir + '\'student marks details.csv\''
The path you end up getting is /user/dev/input/08Aug2017'student marks details.csv', instead of the correct /user/dev/input/08Aug2017/student marks details.csv.
The subprocess call just requires plain strings that correspond to the values that you want, and will not process them the same way the shell does.
Solution
In python, joining paths together is best performed by calling os.path.join. So I would suggest to replace these lines:
source_dir = '/user/dev/input/' + today
hdfs_file_path = source_dir + '\'student marks details.csv\''
with the following:
source_dir = os.path.join('/user/dev/input/', today)
hdfs_file_path = os.path.join(source_dir, 'student marks details.csv')
os.path.join takes care to add a single directory separator (/ on Unix, \ on Windows) between its arguments, so you can't accidentally either forget the separator, or add it twice.

New line without using \n in Python

In python, I'm trying to do something to compile c source codes easily, including a line like that ;
import os
os.system("gcc %s -o %s -ansi" %(filename, filename[:-3])
But it gives me the error ;
sh: 2: -o: not found
To check what is wrong I'm trying to do this ;
print("gcc %s -o %s -ansi" %(filename, filename[:-3]))
and it gives me that ;
gcc foo.c
-o foo -ansi
My question is why it occurs and what should I do to prevent it ?

os.system is depreciated and it's recommended to use subprocess instead.
Splitting the filename on "." will ensure it works correctly even if the file extension differ.
And if you're using Python 3 I suggest you use format.
import subprocess
filename = filename.strip()
subprocess.call('gcc {} -o {} -ansi'.format(filename,filename.split(".")[0])

You probably want to strip away the newline from filename:
import os
import subprocess
filename = filename.strip()
subprocess.call(['gcc', filename, '-o', os.path.splitext(filename)[0], '-ansi'])

filename[:-3] will return the whole string apart from the last 3 characters. Evidently, your last 3 characters are .c\n, which is why you have a newline in your formatted string, because filename equates to foo.c\n.
To remedy this I suppose you have two choices:
1) remove the \n from the string with filename.strip() (recommended, especially if you use filename often for these purposes)
2) use filename[:-1] instead of filename

Xcopy with Python

I'm trying to get xcopy working with python to copy files to a remote system. I am using a very simple test example:
import os
src = "C:\<Username>\Desktop\test2.txt"
dst = "C:\Users\<Username>"
print os.system("xcopy %s %s" % (src, dst))
But for some reason when I run this I get:
Invalid number of parameters
4
Running the xcopy directly from the command line works fine. Any ideas?
Thanks

\t is a tab character. I'd suggest using raw strings for windows paths:
src = r"C:\<Username>\Desktop\test2.txt"
dst = r"C:\Users\<Username>"
This will stop python from surprising you by interpreting some of your backslashes as escape sequences.

In addition to using raw string literals, use the subprocess module instead of os.system - it will take care of quoting your arguments properly if they contain spaces. Thus:
import subprocess
src = r'C:\<Username>\Desktop\test2.txt'
dst = r'C:\Users\<Username>'
subprocess.call(['xcopy', src, dst])

Try prefixing your strings with r. So r"C:\<Username>\Desktop\test2.txt". The problem is that a backslash is treated as a special character within strings.

Calling configuration file ID into Linux Command with Date Time from Python

I'm trying to write a script to get the following outputs to a folder (YYYYMMDDHHMMSS = current date and time) using a Linux command in Python, with the ID's in a configutation file
1234_YYYYMMDDHHMMSS.txt
12345_YYYYMMDDHHMMSS.txt
12346_YYYYMMDDHHMMSS.txt
I have a config file with the list of ID's
id1 = 1234
id2 = 12345
id3 = 123456
I want to be able to loop through these in python and incorporate them into a linux command.
Currently, my linux commands are hardcoded in python as such
import subprocess
import datetime
now = datetime.datetime.now()
subprocess.call('autorep -J 1234* -q > /home/test/output/1234.txt', shell=True)
subprocess.call('autorep -J 12345* -q > /home/test/output/12345.txt', shell=True)
subprocess.call('autorep -J 123456* -q > /home/test/output/123456.txt', shell=True)
print now.strftime("%Y%m%d%H%M%S")
The datetime is defined, but doesn't do anything currently, except print it to the console, when I want to incorporate it into the output txt file. However, I want to be able to write a loop to do something like this
subprocess.call('autorep -J id1* -q > /home/test/output/123456._now.strftime("%Y%m%d%H%M%S").txt', shell=True)
subprocess.call('autorep -J id2* -q > /home/test/output/123456._now.strftime("%Y%m%d%H%M%S").txt', shell=True)
subprocess.call('autorep -J id3* -q > /home/test/output/123456._now.strftime("%Y%m%d%H%M%S").txt', shell=True)
I know that I need to use ConfigParser and currently have been this piece written which simply prints the ID's from the configuration file to the console.
from ConfigParser import SafeConfigParser
import os
parser = SafeConfigParser()
parser.read("/home/test/input/ReportConfig.txt")
def getSystemID():
for section_name in parser.sections():
print
for key, value in parser.items(section_name):
print '%s = %s' % (key,value)
print
getSystemID()
But as mentioned in the beggining of the post, my goal is to be able to loop through the ID's, and incorporate them into my linux command while adding the datetime format to the end of the file. I'm thinking all I need is some kind of while loop in the above function in order to get the type of output I want. However, I'm not sure how to call the ID's and the datetime into a linux command.

So far you have most of what you need, you are just missing a few things.
First, I think using ConfigParser is overkill for this. But it's simple enough so lets continue with it. Lets change getSystemID to a generator returning your IDs instead of printing them out, its just a one line change.
parser = SafeConfigParser()
parser.read('mycfg.txt')
def getSystemID():
for section_name in parser.sections():
for key, value in parser.items(section_name):
yield key, value
With a generator we can use getSystemID in a loop directly, now we need to pass this on to the subprocess call.
# This is the string of the current time, what we add to the filename
now = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
# Notice we can iterate over ids / idnumbers directly
for name, number in getSystemID():
print name, number
Now we need to build the subprocess call. The bulk of your problem above was knowing how to format strings, the syntax is described here.
I'm also going to make two notes on how you use subprocess.call. First, pass a list of arguments instead of a long string. This helps python know what arguments to quote so you don't have to worry about it. You can read about it in the subprocess and shlex documentation.
Second, you redirect the output using > in the command and (as you noticed) need shell=True for this to work. Python can redirect for you, and you should use it.
To pick up where I left off above in the foor loop.
for name, number in getSystemID():
# Make the filename to write to
outfile = '/home/test/output/{0}_{1}.txt'.format(number, now)
# open the file for writing
with open(outfile, 'w') as f:
# notice the arguments are in a list
# stdout=f redirects output to the file f named outfile
subprocess.call(['autorep', '-J', name + '*', '-q'], stdout=f)

You can insert the datetime using Python's format instruction.
For example, you could create a new file with the 1234 prefix and the datime stamp like this:
new_file = open("123456.{0}".format(datetime.datetime.now()), 'w+')
I am not sure if I understood what your are looking for, but I hope this helps.