I have the following command
lessc lessc xyz.less > xyz.css
I want to run that command in python for which i have written this code
try:
project_path = settings.PROJECT_ROOT
less_path = os.path.join(project_path, "static\\less")
css_path = os.path.join(project_path, "static\\css")
except Exception as e:
print traceback.format_exc()
less_file = [f for f in os.listdir(less_path) if isfile(join(less_path, f))]
for files in less_file:
file_name = os.path.splitext(files)[0]
cmd = '%s\%s > %s\%s' % (less_path, files, css_path, file_name + '.css')
p = subprocess.Popen(['lessc', cmd], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
but it gives error windowerror 2 cannot find the path specifies
Make sure that 'lessc' is in your path, you could try using the full path to lessc instead.
You don't need to use shell style redirection with Popen like this, check the subprocess.Popen docs
Here is an example of how to do it without shell redirection:
import subprocess
lessc_command = '/path/to/lessc'
less_file_path = '/path/to/input.less'
css_file_path = '/path/to/output.css'
with open(css_file_path, 'w') as css_file:
less_process = subprocess.Popen([lessc_command, less_file_path], stdout=css_file)
less_process.communicate()
Related
i've been trying to run sox in a python script but it can't the output file and gives me [errno2]
def AudioToSpectrogram(self, files, pixel_per_sec, height, width):
file_name = ("tmp_{}.png").format(random.randint(0, 100000))
command = "sox -V0 {} -n remix 1 rate 10k spectrogram -y {} -x {} -X {} -m -r -o {}".format(files, height, width, pixel_per_sex, file_name)
p = Popen(command, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
Output, errors = p.communicate()
If errors:
Print(errors)
Image = Image.open(file_name)
Os.remove(file_name)
Return np.array(image)
This is the error it gives
Exception: [errno2] No such file or Directory: 'tmp_47483.png'
I hope you could give me some pointers since i am still new in this field, thanks in advance!
Assuming tmp_47483.png is in fact being created, the problem is likely that the command is placing the file in a different folder and Python can't find it inside the current working directory. The
# manually set the full path (make sure the backslashes are escaped by putting two each)
file_name = f"C:\\Full\\Path\\To\\File\\tmp_{random.randint(0, 100000}.png"
# use the os module to join the path
base_dir = "C:\\Full\\Path\\To\\File"
file_name = os.path.join(base_dir, f"tmp_{random.randint(0, 100000}.png")
# if you want it to appear in the same folder as your script:
CWD = os.path.dirname(os.path.realpath(__file__)) # mostly fool-proof way of getting a script's Current Working Directory
file_name = os.path.join(CWD, f"tmp_{random.randint(0, 100000}.png")
Try these and see if they help. If not, make sure that command is actually working and outputting a file somewhere.
I'm trying to run a bash subprocess in python script. However, I seem to be getting the error:
"Please provide the filename(s) of one or more FastQ file(s) to launch Trim Galore!
USAGE: 'trim_galore [options] ' or 'trim_galore --help' for more options"
I wish to iterate over a directory and insert the two files I need into the read1 and read2 variables as a pair into the subprocess (hence the use of indexing)
I don't understand why I'm getting the above error! Not sure why it thinks I haven't provided filenames.
Any ideas would be appreciated, thanks.
import os
import subprocess
indir = "/Users/ainefairbrother/Documents/MSc_Project/TORC_data/unzipped_pretrim"
f = 12
num_of_files = 12
while num_of_files <= f:
pair_count = 1
for root, dirs, all_files in os.walk(indir):
read1 = all_files[pair_count]
pair_count += 1
read2 = all_files[pair_count]
print(read1, read2)
pair_count += 1
process = subprocess.run(['trim_galore', '--paired', '--nextera', read1, read2], stdout=subprocess.PIPE, shell=True) #stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) #shell=True) #stdout=PIPE, stderr=STDOUT,
print(process)
if pair_count > num_of_files:
break
How to get a list of files from hdfs (hadoop) directory using python script?
I have tried with following line:
dir = sc.textFile("hdfs://127.0.0.1:1900/directory").collect()
The directory have list of files "file1,file2,file3....fileN". By using the line i got all the content list only.
But i need to get list of file names.
Can anyone please help me to find out this problem?
Thanks in advance.
Use subprocess
import subprocess
p = subprocess.Popen("hdfs dfs -ls <HDFS Location> | awk '{print $8}'",
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
print line
EDIT: Answer without python. The first option can be used to recursively print all the sub-directories as well. The last redirect statement can be omitted or changed based on your requirement.
hdfs dfs -ls -R <HDFS LOCATION> | awk '{print $8}' > output.txt
hdfs dfs -ls <HDFS LOCATION> | awk '{print $8}' > output.txt
EDIT:
Correcting a missing quote in awk command.
import subprocess
path = "/data"
args = "hdfs dfs -ls "+path+" | awk '{print $8}'"
proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
s_output, s_err = proc.communicate()
all_dart_dirs = s_output.split() #stores list of files and sub-directories in 'path'
Why not have the HDFS client do the hard work by using the -C flag instead of relying on awk or python to print the specific columns of interest?
i.e. Popen(['hdfs', 'dfs', '-ls', '-C', dirname])
Afterwards, split the output on new lines and then you will have your list of paths.
Here's an example along with logging and error handling (including for when the directory/file doesn't exist):
from subprocess import Popen, PIPE
import logging
logger = logging.getLogger(__name__)
FAILED_TO_LIST_DIRECTORY_MSG = 'No such file or directory'
class HdfsException(Exception):
pass
def hdfs_ls(dirname):
"""Returns list of HDFS directory entries."""
logger.info('Listing HDFS directory ' + dirname)
proc = Popen(['hdfs', 'dfs', '-ls', '-C', dirname], stdout=PIPE, stderr=PIPE)
(out, err) = proc.communicate()
if out:
logger.debug('stdout:\n' + out)
if proc.returncode != 0:
errmsg = 'Failed to list HDFS directory "' + dirname + '", return code ' + str(proc.returncode)
logger.error(errmsg)
logger.error(err)
if not FAILED_TO_LIST_DIRECTORY_MSG in err:
raise HdfsException(errmsg)
return []
elif err:
logger.debug('stderr:\n' + err)
return out.splitlines()
For python 3:
from subprocess import Popen, PIPE
hdfs_path = '/path/to/the/designated/folder'
process = Popen(f'hdfs dfs -ls -h {hdfs_path}', shell=True, stdout=PIPE, stderr=PIPE)
std_out, std_err = process.communicate()
list_of_file_names = [fn.split(' ')[-1].split('/')[-1] for fn in std_out.decode().readlines()[1:]][:-1]
list_of_file_names_with_full_address = [fn.split(' ')[-1] for fn in std_out.decode().readlines()[1:]][:-1]
use the following :
hdfsdir = r"hdfs://VPS-DATA1:9000/dir/"
filepaths = [ line.rsplit(None,1)[-1] for line in sh.hdfs('dfs','-ls',hdfsdir).split('\n') if len(line.rsplit(None,1))][1:]
for path in filepaths:
print(path)
to get list of hdfs files in a drectory :
hdfsdir = /path/to/hdfs/directory
filelist = [ line.rsplit(None,1)[-1] for line in sh.hdfs('dfs','-ls',hdfsdir).split('\n') if len(line.rsplit(None,1))][1:]
for path in filelist:
#reading data file from HDFS
with hdfs.open(path, "r") as read_file:
#do what u wanna do
data = json.load(read_file)
this list is a list of all files in hdfs directory
filelist = [ line.rsplit(None,1)[-1] for line in sh.hdfs('dfs','-ls',hdfsdir).split('\n') if len(line.rsplit(None,1))][1:]
I want my script to go under a particular file path mentioned in os.walk() and then execute a grep command on all the files under that location and redirect the output to a file. Below is the script I created, but the subprocess executes ls -al command under the current directory but the print statment show me the contents of os.walk. So I need the subprocess to execute the command under the os.walk path as well.
with open('ipaddressifle.out', 'w') as outfile:
for pdir, dir, files in os.walk(r'/Users/skandasa/perforce/projects/releases/portal-7651'):
for items in files:
print(items)
#subprocess.call(['ls', '-al'])
process = subprocess.Popen(['ls', '-al'], shell= True, stdout=outfile, stderr=outfile)
#process = subprocess.Popen(['grep', 'PORTALSHARED','*', '|', 'awk', '-F', '[','{print', '$1}'], shell= True, stdout=outfile, stderr=outfile)
[output, err] = process.communicate()
And is there anyother way apart from adding a cd command to the subprocess call.
You can use os.chdir(path) to change the current working directory.
I reworked your snippet to use subprocess.check_output to call a command and retrieve its stdout. I also use shlex.split(command) to write the command in a single string and split it correctly for Popen.
The script does os.walk(DIRECTORY) and write the output of ls -la in each subdirectory into OUTPUT_FILE:
import os
import shlex
from subprocess import check_output
DIRECTORY = '/tmp'
OUTPUT_FILE = '/tmp/output.log'
with open(OUTPUT_FILE, 'w') as output:
for parent, _, _ in os.walk(DIRECTORY):
os.chdir(parent)
output.write(check_output(shlex.split('ls -al')))
I'm using PDFminer to convert pdf to html file.
Wrong Code:
def pdf2html(filename, path):
outfile_name = filename.split('.')[0] + '.html'
cmd = ['pdf2txt.py', '-o', path + outfile_name, path + filename]
print ' '.join(cmd)
subprocess.call(cmd, shell=True)
filename = "040214_MOOCs.pdf"
path = "/Users/andy/GoogleDrive/Debate/intelligencesquaredus/data/"
pdf2html(filename, path)
The above code is supposed to run "pdf2txt.py -o /Users/andy/GoogleDrive/Debate/intelligencesquaredus/data/040214_MOOCs.html /Users/andy/GoogleDrive/Debate/intelligencesquaredus/data/040214_MOOCs.pdf"
in the shell.
But there's no output(040214_MOOCs.html) using above code. If I run the command in shell, it generates output with no problem.
Then I tried following script and it works, the only difference is using os.system instead of subprocess.call:
def pdf2html(filename, path):
outfile_name = filename.split('.')[0] + '.html'
cmd = ['pdf2txt.py', '-o', path + outfile_name, path + filename]
print ' '.join(cmd)
os.system(' '.join(cmd))
filename = "040214_MOOCs.pdf"
path = "/Users/andy/GoogleDrive/Debate/intelligencesquaredus/data/"
pdf2html(filename, path)
Also, in wrong code, if I set shell=False, the code also works, why that's the case?
Why subprocess doesn't work in this case while os.system works?
Very confusing, need explanation.
This is likely because of shell mismatch. Can you try running your subprocess call without shell=True?