Develop Reference

empty space after stopwords removal and lemmatisation

The text looks like this before processing
0 [It's, good, for, beginners] positive
1 [I, recommend, this, starter, Ukulele, kit., I... positive
After preprocessing with stopword removal and lemmatisation
nlp = spacy.load('en', disable=['ner', 'parser']) # disabling Named Entity Recognition for speed
def cleaning(doc):
txt = [token.lemma_ for token in doc if not token.is_stop]
if len(txt) > 2:
return ' '.join(txt)
brief_cleaning = (re.sub("[^A-Za-z']+", ' ', str(row)).lower() for row in df3['reviewText'])
txt = [cleaning(doc) for doc in nlp.pipe(brief_cleaning, batch_size=5000, n_threads=-1)]
the result came like this
0 ' good ' ' ' ' beginner ' positive
1 ' ' ' recommend ' ' ' ' starter ' ' ukulele ... positive
As you can see, there are lots of ' ' in the result, what caused this? I'm assuming it's the return ' '.join(txt) and re.sub("[^A-Za-z']+", ' ' that caused it, but if I removed the space or use return (txt), it simply won't remove any stopword or carry out lemmatisation.
Will these empty space cause troubles, or are they necessary, because I'm doing bigram and word2vec afterwards.
How can I fix it and have the result returned as ' recommend ' ' starter ' ' ukulele ' ' kit ' ' need ' ' learn ' ' ukulele '?

Remove special characters and replace spaces with "-"

I'm trying to make a directory that is okay to appear in a URL. I want to ensure that it doesn't contain any special characters and replace any spaces with hyphens.
from os.path import join as osjoin
def image_dir(self, filename):
categorydir = ''.join(e for e in str(self.title.lower()) if e.isalnum())
return "category/" + osjoin(categorydir, filename)
It's removing special characters however I'd like use .replace(" ", "-") to swap out spaces with hyphens

The best way is probably to use the slugify function which takes any string as input and returns an URL-compatible one, yours is not an URL but it will do the trick, eg:
>>> from django.utils.text import slugify
>>> slugify(' Joel is a slug ')
'joel-is-a-slug'

Why don't you use the quote function?
import urllib.parse
urlllib.parse.quote(filename.replace(" ", "-"), safe="")

You can create this functions and call remove_special_chars(s) to do it:
def __is_ascii__(c):
return (ord(c) < 128)
def remove_special_chars(s):
output = ''
for c in s:
if (c.isalpha() and __is_ascii__(c)) or c == ' ':
output = output + c
else:
if c in string.punctuation:
output = output + ' '
output = re.sub(' +', ' ', output)
output = output.replace(' ', '-')
return output
It will remove each non-ASCII character and each element in string.punctuation
EDIT:
This function will substitute each element in string.punctuation with a '-', if you want you can substitute ' ' with '' in the else statement to merge the two parts of the string before and after the punctuation element.

Python: match and replace all whitespaces at the beginning of each line

I need to convert text like this:
' 1 white space before string'
' 2 white spaces before string'
' 3 white spaces before string'
Into a:
' 1 white space before string'
' 2 white spaces before string'
' 3 white spaces before string'
Whitespaces between words and at the end of the line should not be matched, only at the beginning. Also, no need to match tabs. Big thx for help

Use re.sub with a callback that performs the actual replacement:
import re
list_of_strings = [...]
p = re.compile('^ +')
for i, l in enumerate(list_of_strings):
list_of_strings[i] = p.sub(lambda x: x.group().replace(' ', ' '), l)
print(list_of_strings)
[' 1 white space before string',
' 2 white spaces before string',
' 3 white spaces before string'
]
The pattern used here is '^ +' and will search for, and replace whitespaces as long as they're at the start of your string.

If you know it's just spaces as leading whitespace, you could do something like this:
l = ' ' * (len(l) - len(l.lstrip())) + l.lstrip()
Not the most efficient though. This would be a bit better:
stripped = l.strip()
l = ' ' * (len(l) - len(stripped)) + stripped
print(l)
It's one way to do it without the re overhead.
For example:
lines = [
' 1 white space before string',
' 2 white spaces before string',
' 3 white spaces before string',
]
for l in lines:
stripped = l.strip()
l = ' ' * (len(l) - len(stripped)) + stripped
print(l)
Output:
1 white space before string
2 white spaces before string
3 white spaces before string

Splitting Strings within an Array

I am writing a program in python that reads in a text file and executes any python commands within it. The commands may be out of order, but each command has a letter ID such as {% (c) print x %}
I've been able to sort all the commands with in the document into an array, in the correct order. My question is, how to i remove the (c), so i can run exec(statement) on the string?
Here is the full example array
[' (a) import random ', ' (b) x = random.randint(1,6) ', ' (c) print x ', ' (d) print 2*x ']
Also, I am very new to python, my first assignment with it.

You can remove the index part, by using substring:
for cmd in arr:
exec(cmd[5:])

Take everything right to the parenthesis and exec:
for cmd in arr:
exec(cmd.split(") ")[-1])

Stripping the command-id prefixes is a good job for a regular expression:
>>> import re
>>> commands = [' (a) import random ', ' (b) x = random.randint(1,6) ', ' (c) print x ', ' (d) print 2*x ']
>>> [re.search(r'.*?\)\s*(.*)', command).group(1) for command in commands]
['import random ', 'x = random.randint(1,6) ', 'print x ', 'print 2*x ']
The meaning of regex components are:
.*?\) means "Get the shortest group of any characters that ends with a closing-parentheses."
\s* means "Zero or more space characters."
(.*) means "Collect all the remaining characters into group(1)."
How this explanation makes it all clear :-)

Since the pattern looks simple and consistent, you could use regex.
This also allows for both (a) and (abc123) as valid IDs.
import re
lines = [
' (a) import random ',
' (b) x = random.randint(1,6) ',
' (c) print x ',
' (d) print 2*x '
]
for line in lines:
print(re.sub(r"^[ \t]+(\(\w+\))", "", line))
Which would output:
import random
x = random.randint(1,6)
print x
print 2*x
If you really only want to match a single letter, then replace \w+ with [a-zA-Z].

You may use a simple regex to omit the first alpha character in braces as:
import re
lst = [' (a) import random ', ' (b) x = random.randint(1,6) ', ' (c) print x ', ' (d) print 2*x ']
for ele in lst:
print re.sub("^ \([a-z]\)", "", ele)

How do I trim whitespace from a string?

How do I remove leading and trailing whitespace from a string in Python?
" Hello world " --> "Hello world"
" Hello world" --> "Hello world"
"Hello world " --> "Hello world"
"Hello world" --> "Hello world"

To remove all whitespace surrounding a string, use .strip(). Examples:
>>> ' Hello '.strip()
'Hello'
>>> ' Hello'.strip()
'Hello'
>>> 'Bob has a cat'.strip()
'Bob has a cat'
>>> ' Hello '.strip() # ALL consecutive spaces at both ends removed
'Hello'
Note that str.strip() removes all whitespace characters, including tabs and newlines. To remove only spaces, specify the specific character to remove as an argument to strip:
>>> " Hello\n ".strip(" ")
'Hello\n'
To remove only one space at most:
def strip_one_space(s):
if s.endswith(" "): s = s[:-1]
if s.startswith(" "): s = s[1:]
return s
>>> strip_one_space(" Hello ")
' Hello'

As pointed out in answers above
my_string.strip()
will remove all the leading and trailing whitespace characters such as \n, \r, \t, \f, space .
For more flexibility use the following
Removes only leading whitespace chars: my_string.lstrip()
Removes only trailing whitespace chars: my_string.rstrip()
Removes specific whitespace chars: my_string.strip('\n') or my_string.lstrip('\n\r') or my_string.rstrip('\n\t') and so on.
More details are available in the docs.

strip is not limited to whitespace characters either:
# remove all leading/trailing commas, periods and hyphens
title = title.strip(',.-')

This will remove all leading and trailing whitespace in myString:
myString.strip()

You want strip():
myphrases = [" Hello ", " Hello", "Hello ", "Bob has a cat"]
for phrase in myphrases:
print(phrase.strip())

This can also be done with a regular expression
import re
input = " Hello "
output = re.sub(r'^\s+|\s+$', '', input)
# output = 'Hello'

Well seeing this thread as a beginner got my head spinning. Hence came up with a simple shortcut.
Though str.strip() works to remove leading & trailing spaces it does nothing for spaces between characters.
words=input("Enter the word to test")
# If I have a user enter discontinous threads it becomes a problem
# input = " he llo, ho w are y ou "
n=words.strip()
print(n)
# output "he llo, ho w are y ou" - only leading & trailing spaces are removed
Instead use str.replace() to make more sense plus less error & more to the point.
The following code can generalize the use of str.replace()
def whitespace(words):
r=words.replace(' ','') # removes all whitespace
n=r.replace(',','|') # other uses of replace
return n
def run():
words=input("Enter the word to test") # take user input
m=whitespace(words) #encase the def in run() to imporve usability on various functions
o=m.count('f') # for testing
return m,o
print(run())
output- ('hello|howareyou', 0)
Can be helpful while inheriting the same in diff. functions.

In order to remove "Whitespace" which causes plenty of indentation errors when running your finished code or programs in Pyhton. Just do the following;obviously if Python keeps telling that the error(s) is indentation in line 1,2,3,4,5, etc..., just fix that line back and forth.
However, if you still get problems about the program that are related to typing mistakes, operators, etc, make sure you read why error Python is yelling at you:
The first thing to check is that you have your
indentation right. If you do, then check to see if you have
mixed tabs with spaces in your code.
Remember: the code
will look fine (to you), but the interpreter refuses to run it. If
you suspect this, a quick fix is to bring your code into an
IDLE edit window, then choose Edit..."Select All from the
menu system, before choosing Format..."Untabify Region.
If you’ve mixed tabs with spaces, this will convert all your
tabs to spaces in one go (and fix any indentation issues).

I could not find a solution to what I was looking for so I created some custom functions. You can try them out.
def cleansed(s: str):
""":param s: String to be cleansed"""
assert s is not (None or "")
# return trimmed(s.replace('"', '').replace("'", ""))
return trimmed(s)
def trimmed(s: str):
""":param s: String to be cleansed"""
assert s is not (None or "")
ss = trim_start_and_end(s).replace(' ', ' ')
while ' ' in ss:
ss = ss.replace(' ', ' ')
return ss
def trim_start_and_end(s: str):
""":param s: String to be cleansed"""
assert s is not (None or "")
return trim_start(trim_end(s))
def trim_start(s: str):
""":param s: String to be cleansed"""
assert s is not (None or "")
chars = []
for c in s:
if c is not ' ' or len(chars) > 0:
chars.append(c)
return "".join(chars).lower()
def trim_end(s: str):
""":param s: String to be cleansed"""
assert s is not (None or "")
chars = []
for c in reversed(s):
if c is not ' ' or len(chars) > 0:
chars.append(c)
return "".join(reversed(chars)).lower()
s1 = ' b Beer '
s2 = 'Beer b '
s3 = ' Beer b '
s4 = ' bread butter Beer b '
cdd = trim_start(s1)
cddd = trim_end(s2)
clean1 = cleansed(s3)
clean2 = cleansed(s4)
print("\nStr: {0} Len: {1} Cleansed: {2} Len: {3}".format(s1, len(s1), cdd, len(cdd)))
print("\nStr: {0} Len: {1} Cleansed: {2} Len: {3}".format(s2, len(s2), cddd, len(cddd)))
print("\nStr: {0} Len: {1} Cleansed: {2} Len: {3}".format(s3, len(s3), clean1, len(clean1)))
print("\nStr: {0} Len: {1} Cleansed: {2} Len: {3}".format(s4, len(s4), clean2, len(clean2)))

If you want to trim specified number of spaces from left and right, you could do this:
def remove_outer_spaces(text, num_of_leading, num_of_trailing):
text = list(text)
for i in range(num_of_leading):
if text[i] == " ":
text[i] = ""
else:
break
for i in range(1, num_of_trailing+1):
if text[-i] == " ":
text[-i] = ""
else:
break
return ''.join(text)
txt1 = " MY name is "
print(remove_outer_spaces(txt1, 1, 1)) # result is: " MY name is "
print(remove_outer_spaces(txt1, 2, 3)) # result is: " MY name is "
print(remove_outer_spaces(txt1, 6, 8)) # result is: "MY name is"

How do I remove leading and trailing whitespace from a string in Python?
So below solution will remove leading and trailing whitespaces as well as intermediate whitespaces too. Like if you need to get a clear string values without multiple whitespaces.
>>> str_1 = ' Hello World'
>>> print(' '.join(str_1.split()))
Hello World
>>>
>>>
>>> str_2 = ' Hello World'
>>> print(' '.join(str_2.split()))
Hello World
>>>
>>>
>>> str_3 = 'Hello World '
>>> print(' '.join(str_3.split()))
Hello World
>>>
>>>
>>> str_4 = 'Hello World '
>>> print(' '.join(str_4.split()))
Hello World
>>>
>>>
>>> str_5 = ' Hello World '
>>> print(' '.join(str_5.split()))
Hello World
>>>
>>>
>>> str_6 = ' Hello World '
>>> print(' '.join(str_6.split()))
Hello World
>>>
>>>
>>> str_7 = 'Hello World'
>>> print(' '.join(str_7.split()))
Hello World
As you can see this will remove all the multiple whitespace in the string(output is Hello World for all). Location doesn't matter. But if you really need leading and trailing whitespaces, then strip() would be find.

One way is to use the .strip() method (removing all surrounding whitespaces)
str = " Hello World "
str = str.strip()
**result: str = "Hello World"**
Note that .strip() returns a copy of the string and doesn't change the underline object (since strings are immutable).
Should you wish to remove all whitespace (not only trimming the edges):
str = ' abcd efgh ijk '
str = str.replace(' ', '')
**result: str = 'abcdefghijk'

I wanted to remove the too-much spaces in a string (also in between the string, not only in the beginning or end). I made this, because I don't know how to do it otherwise:
string = "Name : David Account: 1234 Another thing: something "
ready = False
while ready == False:
pos = string.find(" ")
if pos != -1:
string = string.replace(" "," ")
else:
ready = True
print(string)
This replaces double spaces in one space until you have no double spaces any more