Develop Reference

removing multiple pipes from a list

so i have some data i have been trying to clean up, its a list and it looks like this
a = [\nlondon\n\n18\n\n20\n\n30\n\n\n\n\njapan\n\n6\n\n80\n\n2\n\n\n\n\nSpain]
i have tried to clean it up by doing this
a.replace("\n", "|")
the output turn out like this :
[london||18||20||30||||japan||6||80||2|||Spain]
if i do this:
a.replace("\n","")
i get this:
[london,"", "", 18,"","",20"","",30,"","","",""japan,"",""6,"","",80,"","",2"","","","",Spain]
can anyone explain why i am having multiple pipes, spaces and whats the best way to clean the data.

Assuming that your input is:
s = '\nlondon\n\n18\n\n20\n\n30\n\n\n\n\njapan\n\n6\n\n80\n\n2\n\n\n\n\nSpain'
The issue is that there are multiple '\n' in-between data, therefore just replacing each '\n' with another character (say '|') will give you as many of the new characters as there were '\n'.
The simplest approach is to use str.split() to get the non-blank data:
l = list(s.split())
print(l)
# ['london', '18', '20', '30', 'japan', '6', '80', '2', 'Spain']
or, combine it with str.join(), if you want to have it separated by '|':
t = '|'.join(s.split())
print(t)
# london|18|20|30|japan|6|80|2|Spain

I tried it and got this:
a = ['\nlondon\n\n18\n\n20\n\n30\n\n\n\n\njapan\n\n6\n\n80\n\n2\n\n\n\n\nSpain']
print(a[0].replace("\n", ""))
Output:
london182030japan6802Spain
Could you please clarify the exact input and the expected output? it does not seem correct yet and I have taken some liberties.
If your input was a string you can use split():
a = '\nlondon\n\n18\n\n20\n\n30\n\n\n\n\njapan\n\n6\n\n80\n\n2\n\n\n\n\nSpain'
print(a.split())
Output:
['london', '18', '20', '30', 'japan', '6', '80', '2', 'Spain']

python regex for incomplete decimals numbers

I have a string of numbers which may have incomplete decimal reprisentation
for example
a = '1. 1,00,000.00 1 .99 1,000,000.999'
desired output
['1','1,00,000.00','1','.99','1,000,000.999']
so far i have tried the following 2
re.findall(r'[-+]?(\d+(?:[.,]\d+)*)',a)
which gives
['1', '1,00,000.00', '1', '99', '1,000,000.999']
which makes .99 to 99 which is not desired
while
re.findall(r'[-+]?(\d*(?:[.,]\d+)*)',a)
gives
['1', '', '', '1,00,000.00', '', '', '1', '', '.99', '', '1,000,000.999', '']
which gives undesirable empty string results as well
this is for finding currency values in a string so the commas separators don't have a set pattern or mat not be present at all

My suggestion is to use the regex below:
I've implemented a snippet in python.
import re
a = '1. 1,00,000.00 1 .99 1,000,000.999'
result = re.split('/\.?\d\.?\,?/', a)
print result
Output:
['1', '1,00,000.00', '1', '.99', '1,000,000.999']

You can use re.split:
import re
a = '1. 1,00,000.00 1 .99 1,000,000.999'
d = re.split('(?<=\d)\.\s+|(?<=\d)\s+', a)
Output:
['1', '1,00,000.00', '1', '.99', '1,000,000.999']

This regex will give you your desired output:
([0-9]+(?=\.))|([0-9,]+\.[0-9]+)|([0-9]+)|(\.[0-9]+)
You can test it here: https://regex101.com/r/VfQIJC/6

Pandas way of splitting a string with multiple delimiters

I'm working with a dataframe where I wish to change entries in country column, eg:
'Bolivia (Plurinational State of)' should be 'Bolivia',
'Switzerland17' should be 'Switzerland'
I have defined the following function:
def process(w):
for i in range(len(w)):
if w[i] in ['(', ')', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '&', '/']:
w = w[0:i]
w = ''.join(w).replace(" ", "")
break
return w
which I have then applied to the dataframe using the python apply function.
energy['Country'] = energy['Country'].apply(process)
While I have been able to achieve the desired output, it is not entirely correct. Some entries like
United Kingdom of Great Britain and Northern Ireland and United States of America20 have changed to
UnitedKingdomofGreatBritainandNorthernIreland and UnitedStatesofAmerica .
What am I doing wrong? Also what would be a more effective, concise code to achieve the result?

I could be missing something, but it looks like
replace(" ", "")
is going to remove spaces, which is exactly what is happening with UnitedStatesofAmerica

Python Regex match string between specific string and end character

I am building a file stripper to build a config report, and I have a very very long string as my base data. The following is a very small snippet of it, but it at least illustrates what I'm working with.
Snippet Example: DEFAULT_GATEWAY=192.168.88.1&DELVRY_AGGREGATION_INTERVAL0=1&DELVRY_AGGREGATION_INTERVAL1=1&DELVRY_SCHEDULE0=1&DELVRY_SNI0=192.168.88.158&DELVRY_USE_SSL_TLS1=0&
How would I go about matching the following:
between "DEFAULT_GATEWAY=" and "&"
between "DELVRY_AGGREGATION_INTERVAL0=" and "&"
between "DELVRY_AGGREGATION_INTERVAL1=" and "&"
between "DELVRY_SCHEDULE=" and "&"
between "DELVRY_SNI0=" and "&"
between "DELVRY_USE_SSL_TLS1=" and "&"
and building a dict with it like:
{"DEFAULT_GATEWAY":"192.168.88.1",
"DELVRY_AGGREGATION_INTERVAL0":"1",
"DELVRY_AGGREGATION_INTERVAL1":"1",
"DELVRY_SCHEDULE0":"1",
"DELVRY_SNI0":"0",
"DELVRY_USE_SSL_TLS1":"0"}
?

Here is a way to do it.
In [1]: input = 'DEFAULT_GATEWAY=192.168.88.1&DELVRY_AGGREGATION_INTERVAL0=1&DELVRY_AGGREGATION_INTERVAL1=1&DELVRY_SCHEDULE0=1&DELVRY_SNI0=192.168.88.158&DELVRY_USE_SSL_TLS1=0&'
In [2]: input.split('&')
Out[2]:
['DEFAULT_GATEWAY=192.168.88.1',
'DELVRY_AGGREGATION_INTERVAL0=1',
'DELVRY_AGGREGATION_INTERVAL1=1',
'DELVRY_SCHEDULE0=1',
'DELVRY_SNI0=192.168.88.158',
'DELVRY_USE_SSL_TLS1=0',
'']
In [3]: [keyval.split('=') for keyval in input.split('&') if keyval]
Out[3]:
[['DEFAULT_GATEWAY', '192.168.88.1'],
['DELVRY_AGGREGATION_INTERVAL0', '1'],
['DELVRY_AGGREGATION_INTERVAL1', '1'],
['DELVRY_SCHEDULE0', '1'],
['DELVRY_SNI0', '192.168.88.158'],
['DELVRY_USE_SSL_TLS1', '0']]
In [4]: dict(keyval.split('=') for keyval in input.split('&') if keyval)
Out[4]:
{'DEFAULT_GATEWAY': '192.168.88.1',
'DELVRY_AGGREGATION_INTERVAL0': '1',
'DELVRY_AGGREGATION_INTERVAL1': '1',
'DELVRY_SCHEDULE0': '1',
'DELVRY_SNI0': '192.168.88.158',
'DELVRY_USE_SSL_TLS1': '0'}
Notes
This is the input line
Split by & to get pairs of key-values. Note the last entry is empty
Split each entry by the equal sign while throwing away empty entries
Build a dictionary
Another Solution
In [8]: import urlparse
In [9]: urlparse.parse_qsl(input)
Out[9]:
[('DEFAULT_GATEWAY', '192.168.88.1'),
('DELVRY_AGGREGATION_INTERVAL0', '1'),
('DELVRY_AGGREGATION_INTERVAL1', '1'),
('DELVRY_SCHEDULE0', '1'),
('DELVRY_SNI0', '192.168.88.158'),
('DELVRY_USE_SSL_TLS1', '0')]
In [10]: dict(urlparse.parse_qsl(input))
Out[10]:
{'DEFAULT_GATEWAY': '192.168.88.1',
'DELVRY_AGGREGATION_INTERVAL0': '1',
'DELVRY_AGGREGATION_INTERVAL1': '1',
'DELVRY_SCHEDULE0': '1',
'DELVRY_SNI0': '192.168.88.158',
'DELVRY_USE_SSL_TLS1': '0'}

Split first by '&' to get a list of strings, then by '=', like this:
d = dict(kv.split('=') for kv in line.split('&'))

import re
keys = {"DEFAULT_GATEWAY",
"DELVRY_AGGREGATION_INTERVAL0",
"DELVRY_AGGREGATION_INTERVAL1",
"DELVRY_SCHEDULE0",
"DELVRY_SNI0",
"DELVRY_USE_SSL_TLS1"}
resdict = {}
for k in keys:
pat = '{}([^&])&'.format(k)
mo = re.search(pat, bigstring)
if mo is None: continue # no match
resdict[k] = mo.group(1)
will leave your desired result in resdict, if bigstring is the string you're searching in.
This assumes you know in advance which keys you'll be looking for, and you keep them in a set keys. If you don't know in advance the keys of interest, that's a very different issue of course.

how to correctly split the following string ? - Python

I have the following file to be parsed:
Total Virtual Clients : 10 (1 Machines)
Current Connections : 10
Total Elapsed Time : 50 Secs (0 Hrs,0 Mins,50 Secs)
Total Requests : 337827 ( 6687/Sec)
Total Responses : 337830 ( 6687/Sec)
Total Bytes : 990388848 ( 20571 KB/Sec)
Total Success Connections : 3346 ( 66/Sec)
Total Connect Errors : 0 ( 0/Sec)
Total Socket Errors : 0 ( 0/Sec)
Total I/O Errors : 0 ( 0/Sec)
Total 200 OK : 33864 ( 718/Sec)
Total 30X Redirect : 0 ( 0/Sec)
Total 304 Not Modified : 0 ( 0/Sec)
Total 404 Not Found : 303966 ( 5969/Sec)
Total 500 Server Error : 0 ( 0/Sec)
Total Bad Status : 303966 ( 5969/Sec)
so I have the parsing algorithm to search the files for those values, however, when I do:
for data in temp:
line = data.strip().split()
print line
where temp is my temporary buffer, which contains those values,
I get:
['Total', 'I/O', 'Errors', ':', '0', '(', '0/Sec)']
['Total', '200', 'OK', ':', '69807', '(', '864/Sec)']
['Total', '30X', 'Redirect', ':', '0', '(', '0/Sec)']
['Total', '304', 'Not', 'Modified', ':', '0', '(', '0/Sec)']
['Total', '404', 'Not', 'Found', ':', '420953', '(', '5289/Sec)']
['Total', '500', 'Server', 'Error', ':', '0', '(', '0/Sec)']
and I wanted:
['Total I/O Errors', '0', '0']
['Total 200 OK', '69807', '864']
['Total 30X Redirect', '0', '0']
and so on.
How could I accomplish that?

You could use a regular expression as follows:
import re
rex = re.compile('([^:]+\S)\s*:\s*(\d+)\s*\(\s*(\d+)/Sec\)')
for line in temp:
match = rex.match(line)
if match:
print match.groups()
which will give you:
['Total Requests', '337827', '6687']
['Total Responses', '337830', '6687']
['Total Success Connections', '3346', '66']
['Total Connect Errors', '0', '0']
['Total Socket Errors', '0', '0']
['Total I/O Errors', '0', '0']
['Total 200 OK', '33864', '718']
['Total 30X Redirect', '0', '0']
['Total 304 Not Modified', '0', '0']
['Total 404 Not Found', '303966', '5969']
['Total 500 Server Error', '0', '0']
['Total Bad Status', '303966', '5969']
Note that will only match lines which correspond to "TITLE:NUMBER(NUMBER/Sec)". You can adapt the expression to match the other lines as well.

regular expressions are overkill for parsing your data, but it is a convenient way to express the fixed length fields. For example
for data in temp:
first, second, third = re.match("(.{28}):(.{21})(.*)", data).groups()
...
This means the first field is 28 chars. Skip the ':', next 21 chars is second field, remainder is the 3rd field

Instead of splitting on whitespace, you will need to split based on the other delimiters in your format, it might look something like this:
for data in temp:
first, rest = data.split(':')
second, rest = rest.split('(')
third, rest = rest.split(')')
print [x.strip() for x in (first, second, third)]