Develop Reference

How can I loop or RegEx through a text stream in Python?

I need to look for patterns in huge text files (around 100 billion characters) that I can only viably access through a URL. Before the files got so big, I was just running a for loop through a str input with this function:
def check_text_for_pattern(source_text, substring_size):
substrings_counter = 0
unique_substrings = [""]
is_unique = True
print("Looking for desired patterns within source text")
for x in range(len(source_text)):
substring_candidate = source_text[x - 1: substring_size + x - 1]
***pattern rules***
for y in unique_substrings:
if y == substring_candidate:
is_unique = False
if is_unique:
print("New unique substring found: " + substring_candidate)
unique_substrings[substrings_counter] = substring_candidate
substrings_counter += 1
is_unique = True
return unique_substrings
It is working well, but I can't seem to figure out the right way to loop through data that is not completely loaded from the very start, partially because a data stream has no len property. How can I keep moving through character subsets without missing any in a scenario like that? Also, since there are multiple files to run through now, how do I signal my code that end of file has been reached so it can move on to the next URL?

select and filtered files in directory with enumerating in loop

I have a folder that contains many eof extension files name I want to sort them in ordinary way with python code (as you can see in my example the name of all my files contain a date like:20190729_20190731 and they are just satellite orbital information files, then select and filtered 1th,24th,47th and.... (index ) of files and delete others because I need every 24 days information files( for example:V20190822T225942_20190824T005942) not all days information .for facility I select and chose these information files from first day I need so the first file is found then I should select 24 days after from first then 47 from first or 24 days after second file and so on. I exactly need to keep my desire files as I said and delete other files in my EOF source folder my desire files are like these
S1A_OPER_AUX_POEORB_OPOD_20190819T120911_V20190729T225942_20190731T005942.EOF
S1A_OPER_AUX_POEORB_OPOD_20190912T120638_V20190822T225942_20190824T005942.EOF
.
.
.
Mr Zach Young wrote this code below and I appreciate him so much I never thought some body would help me. I think I'm very close to the goal
the error is
error is print(f'Keeping {eof_file}') I changed syntax but the same error: print(f"Keeping {eof_file}")
enter code here
from importlib.metadata import files
import pprint
items = os.listdir("C:/Users/m/Desktop/EOF")
eof_files = []
for item in items:
# make sure case of item and '.eof' match
if item.lower().endswith('.eof'):
eof_files.append(item)
eof_files.sort(key=lambda fname : fname.split('_')[5])
print('All EOF files, sorted')
pprint.pprint(eof_files)
print('\nKeeping:')
files_to_delete = []
count = 0
offset = 2
for eof_file in eof_files:
if count == offset:
print(f"Keeping: [eof_file]")
# reset count
count = 0
continue
files_to_delete.append(eof_file)
count += 1
print('\nRemoving:')
for f_delete in files_to_delete:
print(f'Removing: [f_delete]')
staticmethod

Here's a top-to-bottom demonstration.
I recommend that you:
Run that script as-is and make sure your print statements match mine
Swap in your item = os.listdir(...), and see that your files are properly sorted
Play with the offset variable and make sure you can control what should be kept and what should be deleted; notice that an offset of 2 keeps every third file because count starts at 0
You might need to play around and experiment to make sure you're happy before moving to the final step:
Finally, swap in your os.remove(f_delete)
#!/usr/bin/env python3
from importlib.metadata import files
import pprint
items = [
'foo_bar_baz_bak_bam_20190819T120907_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120901_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120905_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120902_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120903_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120904_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120906_V2..._SomeOtherDate.EOF',
'bogus.txt'
]
eof_files = []
for item in items:
# make sure case of item and '.eof' match
if item.lower().endswith('.eof'):
eof_files.append(item)
eof_files.sort(key=lambda fname : fname.split('_')[5])
print('All EOF files, sorted')
pprint.pprint(eof_files)
print('\nKeeping:')
files_to_delete = []
count = 0
offset = 2
for eof_file in eof_files:
if count == offset:
print(f'Keeping {eof_file}')
# reset count
count = 0
continue
files_to_delete.append(eof_file)
count += 1
print('\nRemoving:')
for f_delete in files_to_delete:
print(f'Removing {f_delete}')
When I run that, I get:
All EOF files, sorted
['foo_bar_baz_bak_bam_20190819T120901_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120902_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120903_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120904_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120905_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120906_V2..._SomeOtherDate.EOF',
'foo_bar_baz_bak_bam_20190819T120907_V2..._SomeOtherDate.EOF']
Keeping:
Keeping foo_bar_baz_bak_bam_20190819T120903_V2..._SomeOtherDate.EOF
Keeping foo_bar_baz_bak_bam_20190819T120906_V2..._SomeOtherDate.EOF
Removing:
Removing foo_bar_baz_bak_bam_20190819T120901_V2..._SomeOtherDate.EOF
Removing foo_bar_baz_bak_bam_20190819T120902_V2..._SomeOtherDate.EOF
Removing foo_bar_baz_bak_bam_20190819T120904_V2..._SomeOtherDate.EOF
Removing foo_bar_baz_bak_bam_20190819T120905_V2..._SomeOtherDate.EOF
Removing foo_bar_baz_bak_bam_20190819T120907_V2..._SomeOtherDate.EOF

BioPython iterating through sequences from fasta file

I'm new to BioPython and I'm trying to import a fasta/fastq file and iterate through each sequence, while performing some operation on each sequence. I know this seems basic, but my code below for some reason is not printing correctly.
from Bio import SeqIO
newfile = open("new.txt", "w")
records = list(SeqIO.parse("rosalind_gc.txt", "fasta"))
i = 0
dna = records[i]
while i <= len(records):
print (dna.name)
i = i + 1
I'm trying to basically iterate through records and print the name, however my code ends up only printing "records[0]", where I want it to print "records[1-10]". Can someone explain why it ends up only print "records[0]"?

The reason for your problem is here:
i = 0
dna = records[i]
Your object 'dna' is fixed to the index 0 of records, i.e., records[0]. Since you are not calling it again, dna will always be fixed on that declaration. On your print statement within your while loop, use something like this:
while i <= len(records):
print (records[i].name)
i = i + 1
If you would like to have an object dna as a copy of records entries, you would need to reassign dna to every single index, making this within your while loop, like this:
while i <= len(records):
dna = records[i]
print (dna.name)
i = i + 1
However, that's not the most efficient way. Finally, for you to learn, a much nicer way than with your while loop with i = i + 1 is to use a for loop, like this:
for i in range(0,len(records)):
print (records[i].name)
For loops do the iteration automatically, one by one. range() will give a set of integers from 0 to the length of records. There are also other ways, but I'm keeping it simple.

Breaking a loop

Hello im trying to make an mp3 player. At the moment Im just trying to get it to print the found mp3 file to the console but its stuck in an infinate loop and Im not sure how to break it as im still new to python
def mp3_finder():
times_done = 5
starting = 0
for file in os.listdir(full_dir):
if file.endswith(".mp3"):
while starting < times_done:
starting = starting + 1
print(file)
return mp3_finder()
EDIT:
Sorry i wasnt very clear but what im trying to do is find the mp3 file and print the name to the console 5 times but because it keeps finding the file it keeps printing it to the console until python stops it because it printed hundreds of it

You're calling the function again in the return statement; since you're printing within the function, you can just remove the return entirely.
def mp3_finder():
times_done = 5
starting = 0
for file in os.listdir(full_dir):
if file.endswith(".mp3"):
while starting < times_done:
starting = starting + 1
print(file)
That answers your question about breaking a loop, but perhaps you should ask another one about your code, because I don't think it's going to give you the output you want.

First of all you should probably not call mp3_finder at the end of the function - it will recurse infinitly. Also you probably don't want the inner loop, it wil just print the first file five times. Combined the result will be that the function prints the first file five times, then it calls itself which again prints the first file five times and so on until you reach maximum recursion depth.
What you want to return isn't clear maybe it's OK to just return None (ie skip the return statement entirely). Second you'll need to break out of the loop when you're done.
def mp3_finder():
times_done = 5
starting = 0
for file in os.listdir(full_dir):
if file.endswith(".mp3"):
if starting < times_done:
starting = starting + 1
print(file)
else:
break

def mp3_finder():
times_done = 5
starting = 0
for file in os.listdir(full_dir):
if file.endswith(".mp3"):
while starting < times_done:
starting = starting + 1
print(file)
mp3_finder()
Watch your indent otherwise looks good

Python 'int' object is not subscriptable

Im trying to read a file and make sure that each value is in order. I dont think im converting the string into the integer correctly. Here is some of my code. I am also trying to use flags.
fileName = input("What file name? ")
infile = open(fileName,'r')
correct_order_flag = False
i = 0
line = infile.readline()
while line !="":
for xStr in line.split(" "):
if eval(xStr) [i] < i:
correct_order_flag = True
else:
correct_order_flag = False
i = i+1
if correct_order_flag:
print("Yes, the numbers were in order")
else:
print("No, the numbers were not in order")
count = i - 1
print("There were", count, "numbers.")

You are correct - you are indicating with eval(xStr)[i] that eval(xStr) is an array, and thus can be subscripted. What it looks like you may want (since you say you want to convert the string to an int) is just int(xStr), to make that whole line:
if int(xStr) < i:

For starters, you don't read the whole file at all. Try this:
with open(fileName) as f:
for line in f:
# here goes your code
Not sure though, what do you mean by "each value is in order", but using eval() is a VERY bad idea for any purpose.

I would like to add that because you are comparing xstr[i] to i that unless your first number is less than zero the flag will change, meaning that the sequence 1 2 3 4 5 would print out saying "NO, the numbers were not in order"

As Chris indicated, int(s) is the preferred way to convert a string to an integer. eval(s) is too broad and can be a security risk when evaluating data from an untrusted source.
In addition, there is another error in the script. The *correct_order_flag* is being set on every iteration, so one entry with incorrect order can be masked by a subsequent entry in the correct order. Accordingly, you should break out of the loop when incorrect ordering is found.