Develop Reference

How to cycle through indices

so in this script I am writing to learn python, I would like to just put a wildcard instead of rewriting this whole block just to change line 2. what would be the most efficient way to consolidate this into a loop, where it will just use all d.entries[0-99].content and repeat until finished? if, while, for?
also my try /except does not perform as expected
what gives?
import feedparser, base64
from urlextract import URLExtract
d = feedparser.parse('https://www.reddit.com/r/PkgLinks.rss')
print (d.entries[3].title)
sr = str(d.entries[3].content)
spl1 = sr.split("<p>")
ss = str(spl1)
spl2 = ss.split("</p>")
try:
st = str(spl2[0])
# print(st)
except:
binascii.Error
st = str(spl2[1])
print(st)
#st = str(spl2[0])
spl3 =st.split("', '")
stringnow=str(spl3[1])
b64s1 = stringnow.encode('ascii')
b64s2 = base64.b64decode(b64s1)
stringnew = b64s2.decode('ascii')
print(stringnew)
## but line 15 does nothing, how to fix and also loop through all d.entries[?].content

The loop part is done simply by doing the following"
import feedparser, base64
from urlextract import URLExtract
d = feedparser.parse('https://www.reddit.com/r/PkgLinks.rss')
# loop from 0 to 99
# range(100) goes from 0 and up to and not including 100
for i in range(100):
print (d.entries[i].title)
sr = str(d.entries[i].content)
<< the rest of your code here>>
The data returned from d.entries[i].content is a dictionary but you are converting to a string so you may want to see if you are doing what you really want too. Also when you use .split() it produces a list of the split items but you convert to a string once again (a few time). You may want to relook at that part of the code.

I haven't used regex much but decided to just to play and got this to work. I retrieved the contents of the 'value' key from the dictionary. Then used regex to get the base64 info. I only tried it for the first 5 items (i.e., I changed range(100) to range(5). Hope it helps. If not, I enjoyed doing this. Oh, I left all of the print statements I used as I was working down the code.
import feedparser, base64
from urlextract import URLExtract
import re
d = feedparser.parse('https://www.reddit.com/r/PkgLinks.rss')
for i in range(100):
print (d.entries[i].title)
# .contents is a list.
# print("---------")
# print (type(d.entries[i].content))
print (d.entries[i].content)
print("---------")
# gets the contents of key 'value' in the dictionary that is the 1st item in the list.
string_value = d.entries[3].content[0]['value']
print(string_value)
print("---------")
# this assumes there is always a space between the 1st </p> and the 2nd <p>
# grabs text between using re.search
pattern = "<p>(.*?)</p>"
substring = re.search(pattern, string_value).group(1)
print(substring)
print("---------")
print("---------")
print("---------")
# rest of your code here

how to invert pandas data into text data

I want to make a simple spell corrector system and I have a datafarme like this:
incorrect_word, correct_word
scoohl,school
watn,want
frienf,friend
"I watn to go scoohl"
I want to correct this sentence by replacing the incorrect sample in "incorrect_word" column with the correct sample in the "correct_word" column (if it exists)
how can i do this?
the sample code i wrote and does not work.
text = " شما رفتین مدرسه شون گفتین دستاشون رو بشورن"
# if "دستاشون رو" in text:
# print("yes")
from hazm import *
import pandas as pd
from src.config.config import *
# letters = word_tokenize(text)
# for text in word_tokenize(text):
# print(text)
df = pd.read_excel(FILL_DATA).astype(str)
text = str(text)
for idx, item in enumerate(df['informal']):
if item in text:
text = text.replace(item, df['formal1'].iloc[idx])
# item = item.replace(df['informal'].iloc[idx], df['formal1'].iloc[idx])
print(text)

I would do like this :
df = pd.DataFrame([['scoohl','school'], ['watn','want'], ['frienf','friend']], columns=['incorrect_word', 'correct_word'])
df.index = df['incorrect_word']
df.drop(columns=['incorrect_word'], inplace=True)
text_to_correct = "I watn to go scoohl"
words = text_to_correct.split(' ')
for c, w in enumerate(words):
if w in df.index:
words[c] = df.at[w,'correct_word']
words = ' '.join(words)
words
result :
'I want to go school'

Hello this is very basic python you can do it in this way
df['incorrect']=[x for x in df['Correct'] if len(x)>2]
You should search about lambda, list comprenhension, apply and map
Thank you.

How to remove strings occurring between two strings say 'stringx' and 'stringy' which could occur multiple times in a dataframe

df is unstructured with no columns and rows header. Every columns have strings in which there is a set of pattern which needs to be removed, the pattern is mentioned below as:
Input to one columns of unstructured df as strings:
I am to be read ===start=== I am to be removed ===stop=== I have to be read again ===start=== remove me again ===stop=== continue reading
Ouput needed:
I am to be read I have to be read again continue reading
Here I have to remove from string '===start===' to '===stop===' whenever it occurs. The df has thousands of entries. What is the most efficient way of using regex?
The code below works on a column but takes a long time to complete.
Is there a solution using regex that is most efficient/least time complexity?
df = pd.read_excel("sample_excel.xlsx", header=None)
def removeString(df):
inf = df[0][1]
infcopy = ''
bol = False
start = '*start*'
end = '*stop*'
inf.replace('* start *',start) #in case black space between start
inf.replace('* stop *',end) #in case black space between start
for i in range(len(inf)):
if inf[i] == "*" and inf[i:i+len(start)] == start:
bol = True
if inf[i] == '*' and inf[i+1-len(end):i+1] == end:
bol = False
continue
if bol == False:
infcopy += inf[i]
df[0][1] = infcopy

I think it could look something like this.
import pandas as pd
import re
def removeString(df):
pattern = r'(?:start(.*?)stop)'
df[ColToRemove] = df[ColToRemove].apply(lambda x: re.sub(pattern, "",x))
E.g.
df = pd.DataFrame({'Col1':['startjustsomethingherestop']})
Output:
Col1
0 startjustsomethingherestop
And then,
pattern = r'(?:start(.*?)stop)'
df['Col1'] = df['Col1'].apply(lambda x: re.sub(pattern, "", x))
Output:
Col1
0
The regex pattern defined here will remove everything whenever a match is found for a string that begins with "start" and ends with "stop" and leave you its as the output

Python Programming Error for DataScience DataFrame

I am reading my data from a CSV file using pandas and it works well with range 700. But as soon as I go above 700 and trying to append to a list in python it is showing me list index out of range. But the CSV has around 500K of rows
Can anyone help me with that why is it happening?
Thanks in advance.
import pandas as pd
df_email = pd.read_csv('emails.csv',nrows=800)
test_email = df_email.iloc[:,-1]
list_of_emails = []
for i in range(len(test_email)):
var_email = test_email[i].split("\n") #this code takes one single email splits based on a new line giving a python list of all the strings in the email
email = {}
message_body = ''
for _ in var_email:
if ":" in _:
var_sentence = _.split(":") #this part actually uses the ":" to find the elements in the list that have ":" present
for j in range(len(var_sentence)):
if var_sentence[j].lower().strip() == "from":
email['from'] = var_sentence[var_sentence.index(var_sentence[j+1])].lower().strip()
elif var_sentence[j].lower().strip() == "to":
email['to'] = var_sentence[var_sentence.index(var_sentence[j+1])].lower().strip()
elif var_sentence[j].lower().strip() == 'subject':
if var_sentence[var_sentence.index(var_sentence[j+1])].lower().strip() == 're':
email['subject'] = var_sentence[var_sentence.index(var_sentence[j+2])].lower().strip()
else:
email['subject'] = var_sentence[var_sentence.index(var_sentence[j+1])].lower().strip()
elif ":" not in _:
message_body += _.strip()
email['body'] = message_body
list_of_emails.append(email)

I am not sure of what you are trying to say here (might as well put example inputs and outputs here), but I came across this problem, which might be of the same nature, sometime weeks ago.
CSV files are comma-separated, which means it always takes note of every comma in a line to separate them in columns. If some dirty input from strings in your CSV file are present, then it will mess up the columns that you are expecting to have.
Best solution here is have some code to cleanup your CSV file, change its delimiter to another character (probably '|', '&', or anything that also doesn't mess up with the data), and revise your code to reflect these changes to the CSV.

use the pandas library to read the file.
it is very efficient and saves you time in writing the code yourself.
eg :
import pandas as pd
training_data = pd.read_csv( "train.csv", sep = ",", header = None )

Delete tab character from config files

I am using this function to read a config file.
import numpy as np
stream = np.genfromtxt(filepath, delimiter = '\n', comments='#', dtype= 'str')
It works pretty well but I have a problem: the tab character.
I.e.
output
['\tvalue1 ', ' 1'] ['\t'] ['value2 ', ' 2']
Is there a way to ignore this special char?
My solution is something like that: (It works for my purposes but it's a bit "ugly")
result = {}
for el in stream:
row = el.split('=',1)
try:
if len(row) == 2:
row[0] = row[0].replace(' ','').replace('\t','') #clean the elements from not needed spaces
row[1] = row[1].replace(' ','').replace('\t','')
result[row[0]] = eval(row[1])
except:
print >> sys.stderr,"FATAL ERROR: '"+filepath+"' missetted"
logging.exception(sys.stderr)
sys.exit('')

To replace the tabs with nothing:
stream = [x.replace('\t','') for x in stream]
Or to replace tabs with a single space, and then remove duplicate spaces:
stream = [' '.join(x.replace('\t',' ').split()) for x in stream]
To remove empty strings (source):
stream = filter(None, stream)

There docent seem to be a way to assign multiple delimiters or comments using numpys genfromtext. I would recommend looking elsewhere. Try https://docs.python.org/2/library/configparser.html. Here's a link with a quick example so you can get a feel for how to work with the module https://wiki.python.org/moin/ConfigParserExamples