Output in stdout differs in cmd and Python console - python

I'm new to Python and working on a little program that copies all files of given extension from a folder and it's subfolders to an another directory. Recently I added a simple progress bar and a counter of remaining files.
The problem is that when I run it from cmd and counter comes from say 1000 to 999 cmd adds a zero in the place of a last digit instead of space. Moreover, when the program is finished remaining files counter should be substituted by the word "Done." and it also doesn't work well.
I tried to replace sys.stdout.write with print and tried not to use f-strings, the result is the same.
def show_progress_bar(total, counter=0, length=80):
percent = round(100 * (counter / total))
filled_length = int(length * counter // total)
bar = '=' * filled_length + '-' * (length - filled_length)
if counter < total:
suffix = f'Files left: {total - counter}'
else:
suffix = 'Done.'
sys.stdout.write(f'\rProgress: |{bar}| {percent}% {suffix}')
sys.stdout.flush()
def selective_copy(source, destination, extension):
global counter
show_progress_bar(total)
for foldername, subfolders, filenames in os.walk(source):
for filename in filenames:
if filename.endswith(extension):
if not os.path.exists(os.path.join(destination, filename)):
shutil.copy(os.path.join(foldername, filename), os.path.join(destination, filename))
else:
new_filename = f'{os.path.basename(foldername)}_{filename}'
shutil.copy(os.path.join(foldername, filename), os.path.join(destination, new_filename))
counter += 1
show_progress_bar(total, counter)
I expected that the output in cmd will be the same as in the console, which is this:
Program running:
Progress: |=========-----------------------------------------------------------------------| 12% Files left: 976
Program finished:
Progress: |================================================================================| 100% Done.
But in the cmd I got this:
Program running:
Progress: |=========-----------------------------------------------------------------------| 12% Files left: 9760
Program finished:
Progress: |================================================================================| 100% Done. left: 100

Typically, printing "\r" will return the cursor to the beginning of the line, but it won't erase anything already written. So if you write "1000" followed by "\r" followed by "999", the last 0 of "1000" will still be visible.
(I'm not sure why this isn't happening in your Python console. Maybe it interprets "\r" in a different way. Hard to say without knowing exactly what software you're running.)
One solution is to print a couple of spaces after your output to ensure that slightly longer old messages get overwritten. You can probably get away with just one space for your "Files left:" suffix, since that only decreases by one character at most, but the "done" suffix will need more.
if counter < total:
suffix = f'Files left: {total - counter} '
else:
suffix = 'Done. '

Related

Python strange "string indices must be integers" error

Problem solved! was newfilename[0,3] instead of newfilename[0: 3]
I know this question has been asked before and I have look around on all the answers and the types of problems people have been having related to this error message, but was unable to find anyone with the same type of problem.
I am sowing the whole method just in case. So here is my problem;
When I am trying to get is a substring of "newfilename" using newfilename[int, int] and the compiler keeps thinking I don't have an integer there when I do, at least from my checking I do.
What I'm doing with this code: I am cutting of the end of a filename such as 'foo.txt' to get 'foo' that is saved as newfilename. Then I am adding the number (converted to a string) to the end of it to get 'foo 1' and after that adding back the '.txt' to get the final result of 'foo 1.txt'. The problem occurs when I try to get the substring out and delete the last four characters of the filename to get just 'foo'. After that, I do another check to see if there is a file like that still in the folder and if so I do another set of cutting and pasting to add 1 to the previous file. To be honest, I have not tested of the while loop will work I just thought it should work technically, but my code does not reach that far because of this error lol.
My error:
File "C:/Users/Reaper/IdeaProjects/Curch Rec Managment/Setup.py", line 243, in moveFiles
print(newfilename[0, 3])
TypeError: string indices must be integers
NOTE this error is from when I tried to hard code the numbers it to see if it would work
Here is the current error with the hard code commented out:
newfilename = newfilename[0, int(newfilename.__len__() - 4)] + " 1.m4a"
TypeError: string indices must be integers
What I have tried: I have tried hard coding the numbers is by literally typing in newfilename[0, 7] and still got the same error. I have tried doing this in a separate python file and it seems to work there fine. Also, what is really confusing me is that it works in another part of my program just fine as shown here:
nyear = str(input("Enter new Year: "))
if nyear[0:2] != "20" or nyear.__len__() > 4:
print("Sorry incorrect year. Please try again")
So I have been at it for a while now trying to figure out what in the world is going on and can't get there. Decided I would sleep on it but would post the question just in case. If someone could point out what may be wrong that would be awesome! Or tell me the compilers are just being stupid, well I guess that will do as well.
My function code
def moveFiles(pathList, source, filenameList):
# moves files to new location
# counter keeps track of file name position in list
cnter = 0
for x in pathList:
filename = filenameList[cnter]
#print(x + "\\" + filename)
# new filename
if filename.find("PR") == 0:
newfilename = filename[3:filename.__len__()]
else:
newfilename = filename[2:filename.__len__()]
# checking if file exists and adding numbers to the end if it does
if os.path.isfile(x + "\\" + newfilename):
print("File Name exists!!")
# adding a 1 to the end
print(newfilename)
# PROBLEM ON NEXT TWO LINES, also prob. on any line with the following calls
print(newfilename[0, 3])
newfilename = newfilename[0, int(newfilename.__len__() - 4)] + " 1.m4a"
print("Adding 1:", newfilename)
# once again check if the file exists and adding 1 to the last number
while os.path.isfile(x + "\\" + newfilename):
# me testing if maybe i just can't have math operations withing the substring call
print("File exists again!!")
num = newfilename.__len__() - 6
num2 = newfilename.__len__() - 4
num3 = int(newfilename[num, num2])
num = newfilename.__len__() - 5
newfilename = newfilename[0, num] + str(num3 + 1)
print("Adding 1:", newfilename)
# moving file and deleting prefix
if not os.path.isdir(x):
os.makedirs(x)
os.rename(source + "\\" + filename, x + "\\" + newfilename)
cnter += 1
I think you need this:
print(newfilename[0:3])

fnmatch working inconsistently in different scripts

I am working on a python script that will write input files for an analysis program I use. One of the steps is to take a list of filenames and search the input directory for them, open them, and get some information out of them. I wrote the following using os.walk and fnmatch in a test-script that has the directory of interest hard-coded in, and it worked just fine:
for locus in loci_select: # for each locus we'll include
print("Finding file " + locus)
for root, dirnames, filenames in os.walk('../phylip_wigeon_mid1_names'):
for filename in fnmatch.filter(filenames, locus): # look in the input directory
print("Found file for locus " + locus + " in set")
loci_file = open(os.path.join('../phylip_wigeon_mid1_names/', filename))
with loci_file as f:
for i, l in enumerate(f):
pass
count = (i) * 0.5 # how many individuals present
print(filename + "has sequences for " + str(count) + " individuals")
...and so on (the other bits all work, so I'll spare you).
As soon as I put this into the larger script and switch out the directory names for input arguments, though, it seems to stop working between the third and fourth lines, despite being nearly identical:
for locus in use_loci: # for each locus we'll include
log.info("Finding file " + locus)
for root, dirnames, filenames in os.walk(args.input_dir):
for filename in fnmatch.filter(filenames, locus): # look in the input directory
log.info("Found file for locus " + locus + " in set")
loci_file = open(os.path.join(args.input_dir, filename))
with loci_file as f:
for i, l in enumerate(f):
pass
count = (i) * 0.5 # how many individuals present
log.info(filename + "has sequences for " + str(count) + " individuals")
I've tested it with temporary print statements between the suspected lines, and it seems like they are the culprits, since my screen output looks like:
2015-11-17 15:53:20,505 - write_ima2p_input_file - INFO - Getting selected loci for analysis
2015-11-17 15:53:20,505 - write_ima2p_input_file - INFO - Finding file uce-7999_wigeon_mid1_contigs.phy
2015-11-17 15:53:20,629 - write_ima2p_input_file - INFO - Finding file uce-4686_wigeon_mid1_contigs.phy
2015-11-17 15:53:20,647 - write_ima2p_input_file - INFO - Finding file uce-5012_wigeon_mid1_contigs.phy
...and so on.
I've tried switching out to glob, as well as simple things like rearranging where this section falls in my larger code, but nothing is working. Any insight would be much appreciated!

How to print program working directory in Python 2?

I am writing a Python 2 program to find a file. This program should print each directory it searches at each iteration of the search, but always to the same line in the terminal (i.e. by erasing the text that is already there and moving the cursor to the beginning of the line before printing again.)
This is the code I have so far:
import os
import sys
for root, dirs, files in os.walk("/"):
print root +'\r',
print '\x1b[2K\r',
My problem is that it starts each printout (when it change directory) on a new line; in other words, it doesn't reuse the old line.
How can I ensure all printed output goes to a single line in the terminal?
You need to flush the stdout buffer (depends on the terminal system), and pad the line with whitespace. For example:
for root, dirs, files in os.walk(path):
print "%-80s\r" % (root),
sys.stdout.flush()
time.sleep(1) # For testing
This assumes an arbitrary maximum filename length of 80 characters.
EDIT:
This new solution uses curses, which is part of the standard library:
import curses
import os
import time
win = curses.initscr()
for root, dirs, files in os.walk(path):
win.clear()
win.addstr(0, 0, root)
win.refresh()
time.sleep(1) # For testing purposes
curses.endwin()
This should do it.
for root, dirs, files in os.walk(path):
print '\r', root,
The \r tells python to rewind to the beginning of the current line, like old typewriters.
You might want to pad with spaces to erase the rest of the line, if the current path is shorter than the previous path.
If the text is longer than one line, it will still overflow to the next line.
You need to shorten your output to under the terminal limit.
You could just truncate and put ellipsis at the front:
limit = 30 # for example
message = 'ABCDEFGHIJKLMNOPQRSTUVWX' * 4
if len(message) > limit:
message = '...' + message[-limit+3:]
print message # ...VWXABCDEFGHIJKLMNOPQRSTUVWX
If you want to replace the middle with ..., then you could do:
limit = 30 # for example
message = 'ABCDEFGHIJKLMNOPQRSTUVWX' * 4
length = len(message) # will be 100
if length > limit:
message = list(message)
cut_size = length - limit
start_cut = (length - cut_size) / 2
message[start_cut:start_cut + cut_size + 3] = '...'
message = ''.join(message)
print message # ABCDEFGHIJKLMNO...MNOPQRSTUVWX
Inspired by several ideas from here and there, this works for me well:
import os
import sys
import time # only if you use sleep() function for debugging
top_folder = "/"
max_line_length = 80
for root, dirs, files in os.walk(top_folder):
message = root
# truncate if the path longer than what you want it to be
if len(message) > max_line_length:
message = '[...]' + message[-max_line_length+5:]
# prepare the output string of lenght determined by a variable
output_string = '{0: <' + str(max_line_length) + '}\r' # \r = carret return
# output
print output_string.format(message), # the comma is crucial here
# to see it in action in slow-motion
time.sleep(.4)
The last 2 code lines before the sleep() function line could be combined into one line:
print '{msg: <{width}}\r'.format(msge = message, width = max_line_length),

Automator/Applescript rename files if

I have a large list of images that have been misnamed by my artist. I was hoping to avoid giving him more work by using Automator but I'm new to it. Right now they're named in order what001a and what002a but that should be what001a and what001b. So basically odd numbered are A and even numbered at B. So i need a script that changes the even numbered to B images and renumbers them all to the proper sequential numbering. How would I go about writing that script?
A small Ruby script embedded in an AppleScript provides a very comfortable solution, allowing you to select the files to rename right in Finder and displaying an informative success or error message.
The algorithm renames files as follows:
number = first 3 digits in filename # e.g. "006"
letter = the letter following those digits # e.g. "a"
if number is even, change letter to its successor # e.g. "b"
number = (number + 1)/2 # 5 or 6 => 3
replace number and letter in filename
And here it is:
-- ask for files
set filesToRename to choose file with prompt "Select the files to rename" with multiple selections allowed
-- prepare ruby command
set ruby_script to "ruby -e \"s=ARGV[0]; m=s.match(/(\\d{3})(\\w)/); n=m[1].to_i; a=m[2]; a.succ! if n.even?; r=sprintf('%03d',(n+1)/2)+a; puts s.sub(/\\d{3}\\w/,r);\" "
tell application "Finder"
-- process files, record errors
set counter to 0
set errors to {}
repeat with f in filesToRename
try
do shell script ruby_script & (f's name as text)
set f's name to result
set counter to counter + 1
on error
copy (f's name as text) to the end of errors
end try
end repeat
-- display report
set msg to (counter as text) & " files renamed successfully!\n"
if errors is not {} then
set AppleScript's text item delimiters to "\n"
set msg to msg & "The following files could NOT be renamed:\n" & (errors as text)
set AppleScript's text item delimiters to ""
end if
display dialog msg
end tell
Note that it will fail when the filename contains spaces.
A friend of mine wrote a Python script to do what I needed. Figured I'd post it here as an answer for anyone stumbling upon a similar problem looking for help. It is in Python though so if anyone wants to convert it to AppleScript for those that may need it go for it.
import os
import re
import shutil
def toInt(str):
try:
return int(str)
except:
return 0
filePath = "./"
extension = "png"
dirList = os.listdir(filePath)
regx = re.compile("[0-9]+a")
for filename in dirList:
ext = filename[-len(extension):]
if(ext != extension): continue
rslts = regx.search(filename)
if(rslts == None): continue
pieces = regx.split(filename)
if(len(pieces) < 2): pieces.append("")
filenumber = toInt(rslts.group(0).rstrip("a"))
newFileNum = (filenumber + 1) / 2
fileChar = "b"
if(filenumber % 2): fileChar = "a"
newFileName = "%s%03d%s%s" % (pieces[0], newFileNum, fileChar, pieces[1])
shutil.move("%s%s" % (filePath, filename), "%s%s" % (filePath, newFileName))

Print to the same line and not a new line? [duplicate]

This question already has answers here:
How to overwrite the previous print to stdout?
(18 answers)
Why doesn't print output show up immediately in the terminal when there is no newline at the end?
(1 answer)
Closed last month.
Basically I want to do the opposite of what this guy did... hehe.
Python Script: Print new line each time to shell rather than update existing line
I have a program that is telling me how far along it is.
for i in some_list:
#do a bunch of stuff.
print i/len(some_list)*100," percent complete"
So if len(some_list) was 50, I'd get that last line printed 50 times over. I want to print one line and keep updating that line. I know I know this is probably the lamest question you'll read all day. I just can't figure out the four words I need to put into google to get the answer.
Update! I tried mvds' suggestion which SEEMED right. The new code
print percent_complete," \r",
Percent complete is just a string (I was abstracting the first time now I an trying to be literal). The result now is that it runs the program, doesn't print ANYTHING until after the program is over, and then prints "100 percent complete" on one and only one line.
Without the carriage return (but with the comma, half of mvds' suggestion) it prints nothing until the end. And then prints:
0 percent complete 2 percent complete 3 percent complete 4 percent complete
And so on. So now the new issue is that with the comma it doesn't print until the program is finished.
With the carriage return and no comma it behaves the exact same as with neither.
It's called the carriage return, or \r
Use
print i/len(some_list)*100," percent complete \r",
The comma prevents print from adding a newline. (and the spaces will keep the line clear from prior output)
Also, don't forget to terminate with a print "" to get at least a finalizing newline!
From python 3.x you can do:
print('bla bla', end='')
(which can also be used in Python 2.6 or 2.7 by putting from __future__ import print_function at the top of your script/module)
Python console progressbar example:
import time
# status generator
def range_with_status(total):
""" iterate from 0 to total and show progress in console """
n=0
while n<total:
done = '#'*(n+1)
todo = '-'*(total-n-1)
s = '<{0}>'.format(done+todo)
if not todo:
s+='\n'
if n>0:
s = '\r'+s
print(s, end='')
yield n
n+=1
# example for use of status generator
for i in range_with_status(10):
time.sleep(0.1)
For me, what worked was a combo of Remi's and siriusd's answers:
from __future__ import print_function
import sys
print(str, end='\r')
sys.stdout.flush()
In Python 3.3+ you don’t need sys.stdout.flush(). print(string, end='', flush=True) works.
So
print('foo', end='')
print('\rbar', end='', flush=True)
will overwrite ‘foo’ with ‘bar’.
for Console you'll probably need
sys.stdout.flush()
to force update. I think using , in print will block stdout from flushing and somehow it won't update
Late to the game - but since the none of the answers worked for me (I didn't try them all) and I've come upon this answer more than once in my search ... In python 3, this solution is pretty elegant and I believe does exactly what the author is looking for, it updates a single statement on the same line. Note, you may have to do something special if the line shrinks instead of grows (like perhaps make the string a fixed length with padded spaces at the end)
if __name__ == '__main__':
for i in range(100):
print("", end=f"\rPercentComplete: {i} %")
time.sleep(0.2)
As of end of 2020 and Python 3.8.5 on linux console for me only this works:
print('some string', end='\r')
Credit goes to: This post
If you are using Spyder, the lines just print continuously with all the previous solutions. A way to avoid that is using:
for i in range(1000):
print('\r' + str(round(i/len(df)*100,1)) + '% complete', end='')
sys.stdout.flush()
This works for me, hacked it once to see if it is possible, but never actually used in my program (GUI is so much nicer):
import time
f = '%4i %%'
len_to_clear = len(f)+1
clear = '\x08'* len_to_clear
print 'Progress in percent:'+' '*(len_to_clear),
for i in range(123):
print clear+f % (i*100//123),
time.sleep(0.4)
raw_input('\nDone')
As of 2021, for Python 3.9.0 the following solution worked for me in Windows 10, Pycharm.
print('\r some string ', end='', flush=True)
import time
import sys
def update_pct(w_str):
w_str = str(w_str)
sys.stdout.write("\b" * len(w_str))
sys.stdout.write(" " * len(w_str))
sys.stdout.write("\b" * len(w_str))
sys.stdout.write(w_str)
sys.stdout.flush()
for pct in range(0, 101):
update_pct("{n}%".format(n=str(pct)))
time.sleep(0.1)
\b will move the location of the cursor back one space
So we move it back all the way to the beginning of the line
We then write spaces to clear the current line - as we write spaces the cursor moves forward/right by one
So then we have to move the cursor back at the beginning of the line before we write our new data
Tested on Windows cmd using Python 2.7
Try it like this:
for i in some_list:
#do a bunch of stuff.
print i/len(some_list)*100," percent complete",
(With a comma at the end.)
For Python 3+
for i in range(5):
print(str(i) + '\r', sep='', end ='', file = sys.stdout , flush = False)
In those cases, with python 3.x, I'm using the following code:
for ii in range(100):
print(f"\rPercent: {ii+1} %", end=" "*20)
The problem with some other answers is that if your printed string goes shorter at one step, the last characters from the previous string won't be overwrited.
So I use end=" "*20 in order to overwrite the previous line with whitespace. Just make sure that 20 is longer than the length of your longest string.
Based on Remi answer for Python 2.7+ use this:
from __future__ import print_function
import time
# status generator
def range_with_status(total):
""" iterate from 0 to total and show progress in console """
import sys
n = 0
while n < total:
done = '#' * (n + 1)
todo = '-' * (total - n - 1)
s = '<{0}>'.format(done + todo)
if not todo:
s += '\n'
if n > 0:
s = '\r' + s
print(s, end='\r')
sys.stdout.flush()
yield n
n += 1
# example for use of status generator
for i in range_with_status(50):
time.sleep(0.2)
For Python 3.6+ and for any list rather than just ints, as well as using the entire width of your console window and not crossing over to a new line, you could use the following:
note: please be informed, that the function get_console_with() will work only on Linux based systems, and as such you have to rewrite it to work on Windows.
import os
import time
def get_console_width():
"""Returns the width of console.
NOTE: The below implementation works only on Linux-based operating systems.
If you wish to use it on another OS, please make sure to modify it appropriately.
"""
return int(os.popen('stty size', 'r').read().split()[1])
def range_with_progress(list_of_elements):
"""Iterate through list with a progress bar shown in console."""
# Get the total number of elements of the given list.
total = len(list_of_elements)
# Get the width of currently used console. Subtract 2 from the value for the
# edge characters "[" and "]"
max_width = get_console_width() - 2
# Start iterating over the list.
for index, element in enumerate(list_of_elements):
# Compute how many characters should be printed as "done". It is simply
# a percentage of work done multiplied by the width of the console. That
# is: if we're on element 50 out of 100, that means we're 50% done, or
# 0.5, and we should mark half of the entire console as "done".
done = int(index / total * max_width)
# Whatever is left, should be printed as "unfinished"
remaining = max_width - done
# Print to the console.
print(f'[{done * "#"}{remaining * "."}]', end='\r')
# yield the element to work with it
yield element
# Finally, print the full line. If you wish, you can also print whitespace
# so that the progress bar disappears once you are done. In that case do not
# forget to add the "end" parameter to print function.
print(f'[{max_width * "#"}]')
if __name__ == '__main__':
list_of_elements = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j']
for e in range_with_progress(list_of_elements):
time.sleep(0.2)
If you are using Python 3
then this is for you and it really works.
print(value , sep='',end ='', file = sys.stdout , flush = False)
Just figured this out on my own for showing a countdown but it would also work for a percentage.
import time
#Number of seconds to wait
i=15
#Until seconds has reached zero
while i > -1:
#Ensure string overwrites the previous line by adding spaces at end
print("\r{} seconds left. ".format(i),end='')
time.sleep(1)
i-=1
print("") #Adds newline after it's done
As long as whatever comes after '/r' is the same length or longer (including spaces) than the previous string, it will overwrite it on the same line. Just make sure you include the end='' otherwise it will print to a newline. Hope that helps!
for object "pega" that provides StartRunning(), StopRunning(),
boolean getIsRunning() and integer getProgress100() returning
value in range of 0 to 100, this provides text progress bar
while running...
now = time.time()
timeout = now + 30.0
last_progress = -1
pega.StartRunning()
while now < timeout and pega.getIsRunning():
time.sleep(0.5)
now = time.time()
progress = pega.getTubProgress100()
if progress != last_progress:
print('\r'+'='*progress+'-'*(100-progress)+' ' + str(progress) + "% ", end='', flush=True)
last_progress = progress
pega.StopRunning()
progress = pega.getTubProgress100()
print('\r'+'='*progress+'-'*(100-progress)+' ' + str(progress) + "% ", flush=True)

Categories

Resources