Develop Reference

How to eliminate null valued cells from a CSV dataset using Python?

Each row represents a person(315 total) and each column represent a choice scenario(16 total). Each person responded randomly to 4 consecutive choice scenarios. I want to have four columns consecutively having the responses of each person and do away with any blank cell.Image of the excel sheet
import pandas as pd
df = pd.read_csv(r"C:\Users\Admin\Desktop\Book2.csv")
for (r,c) in df.iterrows():
if df.iat[r,c] is not None:
for i in range(4):
print(str(df.iat[r,c+i]))
UPDATE
I have managed to get data row wise into a list and grouped them in groups of 4 (as i need it). Now how do I keep the elements with values other than ' '?
import csv
rowdata = []
with open(r'C:\Users\ARPLAB31\Desktop\SPdata.csv') as inputfile:
reader = csv.reader(inputfile)
rowdata = list(reader)
r= []
for i in range(1,718,1):
for j in range(28):
if len(rowdata[i][j])!=0:
r.append(rowdata[i][j])
cardref = [r[x:x+4] for x in range(0, len(r),4)] '''cardref contains the partitioned data.'''
print(cardref)
OUTPUT:
[[' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['BB', 'BB', 'CC', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['CC', 'BB', 'CC', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['CC', 'CC', 'CC', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['CC', 'CC', 'AA', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['CC', 'BB', 'CC', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' ']]

Use df.isnull() [ where df is pandas dataframe]
A good resource to find the null value in dataframe in pandas.
https://dzone.com/articles/pandas-find-rows-where-columnfield-is-null

You can read each row, obtain the not null fields, and create a new CSV from there.
For example:
data = ",,2,2,2,2,,,"
arr = filter(None, data.split(",")) #removes null fields
",".join(arr) #"2,2,2,2"

Thanks all of you. I happened to solve my problem with the help of all the above comments. Please mention any changes to the code in the comment.
import pandas
import csv
rowdata = []
''' READING CSV INTO LIST'''
with open('FILE.csv') as inputfile:
reader = csv.reader(inputfile)
rowdata = list(reader)
'''RECORDING THE POSITION OF NON-EMPTY ELEMENTS'''
r= []
for i in range(1,718,1):
for j in range(28):
if len(rowdata[i][j])!=0:
r.append(j)
continue
''' RE-GROUPING LIST AS LIST IN LIST'''
resp_index = [r[x:x+4] for x in range(0, len(r),4)]
print(resp_index)
print(len(resp_index))
'''ELIMINATING BLANK SPACES AND STORING INTO NEW LIST'''
s= []
for i in range(1,718,1):
for j in range(28):
if len(rowdata[i][j])!=0:
s.append(rowdata[i][j])
''' RE-GROUPING LIST AS LIST IN LIST'''
resp_main = [s[x:x+4] for x in range(0, len(s),4)]
print(resp_main)
print(len(resp_main))
pd = pandas.DataFrame(resp_index)
pe = pandas.DataFrame(resp_main)
'''SAVING TO CSV FILES'''
pd.to_csv('INDEX.csv')
pe.to_csv('RESPONSE.csv')

How to get access to every element of the 2d array and than replace them?

arr = [['.' for i in range(4)] for j in range(4)]
for line, i in enumerate(arr):
for column, j in enumerate(i):
print(j, 'at column', column+1, 'line', line+1) # we can know which
# postition takes
# every element
How out of loop to check if coordinate is different with another coordinate.
What I want to get in final:
pseudocode:
#arr[x][y]
arr[1][0] = 'new'
if arr[1][4] - arr[1][0] == 4: # i.e. coord are different by `y` on 4 pos
arr[1][4] = 'new'`
# Before || After
[[' ', 'new', ' ', ' '],|| [[' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ']] || [' ', 'new', ' ', ' ']]
OR
#arr[x][y]
arr[0][0] = 'new'
if arr[3][0] - arr[0][0] == 3: # i.e. coord are different by `x` on 3 pos
arr[3][0] = 'new'`
# Before || After
[['new', ' ', ' ', ' '],|| [[' ', ' ', ' ', 'new'],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ']] || [' ', ' ', ' ', ' ']]
Necessarily need to know which list in main list takes position, but how it does outside of loop without numpy, using native python?

Question: How out of loop to check if coordinate is different with another coordinate.
Define the coordinate's using tuple, then compare the tuples:
RC = (1,4)
RC2 = (1,0)
if RC == RC2:
print('Equal')
else:
print('Different')
Type list is 0 Based
You have a list of lists.
RC Coordinate System, the same as with Excel
R == index of list in list == y == Row
C == index inside a list == x == Column
A B C D
C => 0 1 2 3
----------------------
R:0 | [0.0, 0.1, 0.2, 0.3]
R:1 | [1.0, 1.1, 1.2, 1.3]
R:2 | [2.0, 2.1, 2.2, 2.3]
R:3 | [3.0, 3.1, 3.2, 3.3]
Use the opposite
#arr[y][x]
The first index=y is the index of a list in arr, alias row.
The second index=x is the index inside the list selected with y, alias column.

Handle Fortran Character Arrays from Python with F2PY

I have a legacy Fortran library I've wrapped with F2PY. However, I'm at a loss for how to properly read character arrays declared as module data, from Python. The data data comes through, but the array is transposed in such a way that it is indiscernible. How can I get Numpy to correctly handle my array? I'd be satisfied with a 2 dimensional array of characters if they were in an intelligible order.
The character arrays are declared and populated in Fortran like so:
module plot_mod
implicit none
CHARACTER*4, JSP(39)
...
JSP = (/ &
'SF ', 'WF ', 'GF ', 'AF ', 'RF ', 'SS ', 'NF ', &
'YC ', 'IC ', 'ES ', 'LP ', 'JP ', 'SP ', 'WP ', &
'PP ', 'DF ', 'RW ', 'RC ', 'WH ', 'MH ', 'BM ', &
'RA ', 'WA ', 'PB ', 'GC ', 'AS ', 'CW ', 'WO ', &
'WJ ', 'LL ', 'WB ', 'KP ', 'PY ', 'DG ', 'HT ', &
'CH ', 'WI ', ' ', 'OT '/)
end module plot_mod
In Python 2.7 (previous version of numpy) I could do this:
x = numpy.frombuffer(fvslib.plot_mod.jsp.data, numpy.dtype('a4'))
But now Python (3.4.4) and Numpy (1.10.4) raises an error, BufferError: memoryview: underlying buffer is not C-contiguous.
I know I should be able to get Numpy to handle this for me by reshaping, or using stride tricks, but I can't seem to figure it out. The array is reported as F-contiguous, so at least that seems correct.
If I simply print the array it looks like this:
array([[b'S', b' ', b' ', b'L'],
[b'F', b'L', b' ', b' '],
[b' ', b'P', b'B', b' '],
[b' ', b' ', b'M', b'W'],
[b'W', b' ', b' ', b'B'],
[b'F', b'J', b' ', b' '],
[b' ', b'P', b'R', b' '],
[b' ', b' ', b'A', b'K'],
[b'G', b' ', b' ', b'P'],
[b'F', b'S', b' ', b' '],
[b' ', b'P', b'W', b' '],
[b' ', b' ', b'A', b'P'],
[b'A', b' ', b' ', b'Y'],
[b'F', b'W', b' ', b' '],
[b' ', b'P', b'P', b' '],
[b' ', b' ', b'B', b'D'],
[b'R', b' ', b' ', b'G'],
[b'F', b'P', b' ', b' '],
[b' ', b'P', b'G', b' '],
[b' ', b' ', b'C', b'H'],
[b'S', b' ', b' ', b'T'],
[b'S', b'D', b' ', b' '],
[b' ', b'F', b'A', b' '],
[b' ', b' ', b'S', b'C'],
[b'N', b' ', b' ', b'H'],
[b'F', b'R', b' ', b' '],
[b' ', b'W', b'C', b' '],
[b' ', b' ', b'W', b'W'],
[b'Y', b' ', b' ', b'I'],
[b'C', b'R', b' ', b' '],
[b' ', b'C', b'W', b' '],
[b' ', b' ', b'O', b' '],
[b'I', b' ', b' ', b' '],
[b'C', b'W', b' ', b' '],
[b' ', b'H', b'W', b' '],
[b' ', b' ', b'J', b'O'],
[b'E', b' ', b' ', b'T'],
[b'S', b'M', b' ', b' '],
[b' ', b'H', b'L', b' ']],
dtype='|S1')
What I would like an array like this:
[['SF ']
, ['WF ']
, ['GF ']
, ['AF ']
, ['RF ']
, ['SS ']
, ['NF ']
, ['YC ']
, ['IC ']
, ['ES ']
, ['LP ']
, ['JP ']
, ['SP ']
, ['WP ']
, ['PP ']
, ['DF ']
, ['RW ']
, ['RC ']
, ['WH ']
, ['MH ']
, ['BM ']
, ['RA ']
, ['WA ']
, ['PB ']
, ['GC ']
, ['AS ']
, ['CW ']
, ['WO ']
, ['WJ ']
, ['LL ']
, ['WB ']
, ['KP ']
, ['PY ']
, ['DG ']
, ['HT ']
, ['CH ']
, ['WI ']
, [' ']
, ['OT ']]

I haven't tried running f2py on your module, but if I define the array you show as:
In [11]: s = array([[b'S', b' ', b' ', b'L'],
...: [b'F', b'L', b' ', b' '],
...: [b' ', b'P', b'B', b' '],
...: [b' ', b' ', b'M', b'W'],
...: [b'W', b' ', b' ', b'B'],
...: [b'F', b'J', b' ', b' '],
...: [b' ', b'P', b'R', b' '],
...: [b' ', b' ', b'A', b'K'],
...: [b'G', b' ', b' ', b'P'],
...: [b'F', b'S', b' ', b' '],
...: [b' ', b'P', b'W', b' '],
...: [b' ', b' ', b'A', b'P'],
...: [b'A', b' ', b' ', b'Y'],
...: [b'F', b'W', b' ', b' '],
...: [b' ', b'P', b'P', b' '],
...: [b' ', b' ', b'B', b'D'],
...: [b'R', b' ', b' ', b'G'],
...: [b'F', b'P', b' ', b' '],
...: [b' ', b'P', b'G', b' '],
...: [b' ', b' ', b'C', b'H'],
...: [b'S', b' ', b' ', b'T'],
...: [b'S', b'D', b' ', b' '],
...: [b' ', b'F', b'A', b' '],
...: [b' ', b' ', b'S', b'C'],
...: [b'N', b' ', b' ', b'H'],
...: [b'F', b'R', b' ', b' '],
...: [b' ', b'W', b'C', b' '],
...: [b' ', b' ', b'W', b'W'],
...: [b'Y', b' ', b' ', b'I'],
...: [b'C', b'R', b' ', b' '],
...: [b' ', b'C', b'W', b' '],
...: [b' ', b' ', b'O', b' '],
...: [b'I', b' ', b' ', b' '],
...: [b'C', b'W', b' ', b' '],
...: [b' ', b'H', b'W', b' '],
...: [b' ', b' ', b'J', b'O'],
...: [b'E', b' ', b' ', b'T'],
...: [b'S', b'M', b' ', b' '],
...: [b' ', b'H', b'L', b' ']],
...: dtype='|S1')
I can get an array that looks like what you want with:
In [12]: s.T.reshape(-1, 4).view('S4')
Out[12]:
array([[b'SF '],
[b'WF '],
[b'GF '],
[b'AF '],
[b'RF '],
[b'SS '],
[b'NF '],
[b'YC '],
[b'IC '],
[b'ES '],
[b'LP '],
[b'JP '],
[b'SP '],
[b'WP '],
[b'PP '],
[b'DF '],
[b'RW '],
[b'RC '],
[b'WH '],
[b'MH '],
[b'BM '],
[b'RA '],
[b'WA '],
[b'PB '],
[b'GC '],
[b'AS '],
[b'CW '],
[b'WO '],
[b'WJ '],
[b'LL '],
[b'WB '],
[b'KP '],
[b'PY '],
[b'DG '],
[b'HT '],
[b'CH '],
[b'WI '],
[b' '],
[b'OT ']],
dtype='|S4')
Note that the data type is 'S4', to match the declared size of the Fortran array.
That result leaves a trivial second dimension, so you might want to convert it to a one-dimensional array, e.g.
In [22]: s.T.reshape(-1, 4).view('S4')[:,0]
Out[22]:
array([b'SF ', b'WF ', b'GF ', b'AF ', b'RF ', b'SS ', b'NF ',
b'YC ', b'IC ', b'ES ', b'LP ', b'JP ', b'SP ', b'WP ',
b'PP ', b'DF ', b'RW ', b'RC ', b'WH ', b'MH ', b'BM ',
b'RA ', b'WA ', b'PB ', b'GC ', b'AS ', b'CW ', b'WO ',
b'WJ ', b'LL ', b'WB ', b'KP ', b'PY ', b'DG ', b'HT ',
b'CH ', b'WI ', b' ', b'OT '],
dtype='|S4')

For completeness I'll include this alternative solution. Same results as #Warren Weckesser, but requires an additional import.
from numpy.lib import stride_tricks
spp = stride_tricks.as_strided(jsp, strides=(jsp.shape[1],1))
# View as S4 and strip whitespace
spp = np.char.strip(spp.view('S4'))

Pythonic way of creating list of lists from list

I would like to create a list of lists from a list.
The list looks like this:
level = [' WWWWWWWWWWWWWWWWW', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C E']
I need to create this:
[[' ', ' ', ' ', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'E']]
I have done it this way:
listofLists = []
for row in level:
liss = []
for col in row:
liss.append(col)
listofLists.append(liss)
What is a more pythonic way or shorter way of doing this?

>>> listofLists = map(list,level)
(in python3, if you really need a list, do list(map(list, level)))

When you call list() on a string this will return the list of all its characters (including spaces).
level = [' WWWWWWWWWWWWWWWWW', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C E']
trasnsformed = [list(x) for x in level]

Unique combinations of two lists – comparison at each index position

I've searched pretty thoroughly, and haven't found any info – hopefully I didn't miss anything. I have two lists:
list1 = (a, b, c, d)
list2 = (a, b*, c*, d)
I would like to generate all possible unique 2 list pairings that only look for differences at each index value. For example, the results here would be:
list1_new = (a, b*, c, d)
list2_new = (a, b, c*, d)
Note: I don't care to distinguish between list1 and list2, i.e., list1_new = (a, b*, c*, d) would not be considered unique as it matches the original list2.
I've played around with itertools but haven't been able to figure out how to compare at each index position.
I used small lists for this example, but I actually will have larger lists of 10+ items.

from itertools import product
list1 = ["a ", "b ", "c ", "d ", "e ", "f ", "g "]
list2 = ["a ", "b*", "c*", "d ", "e*", "f*", "g "]
# figure out which offsets have alternable values
crosses = [a != b for a,b in zip(list1, list2)]
offsets = [i for i,cross in enumerate(crosses) if cross]
# decide which offsets will be swapped
basis = [[0,1] for ofs in offsets]
basis[0] = [0] # only the first half - remainder are mirrors
swaps = product(*basis)
next(swaps, None) # skip base state == (list1, list2)
# print all viable swaps
list3 = list1[:]
list4 = list2[:]
for sw in swaps:
# build output lists
for which,offs in zip(sw, offsets):
list3[offs] = [list1, list2][which][offs]
list4[offs] = [list2, list1][which][offs]
# display output
print("\n{}\n{}".format(list3, list4))
gives
['a ', 'b ', 'c ', 'd ', 'e ', 'f*', 'g ']
['a ', 'b*', 'c*', 'd ', 'e*', 'f ', 'g ']
['a ', 'b ', 'c ', 'd ', 'e*', 'f ', 'g ']
['a ', 'b*', 'c*', 'd ', 'e ', 'f*', 'g ']
['a ', 'b ', 'c ', 'd ', 'e*', 'f*', 'g ']
['a ', 'b*', 'c*', 'd ', 'e ', 'f ', 'g ']
['a ', 'b ', 'c*', 'd ', 'e ', 'f ', 'g ']
['a ', 'b*', 'c ', 'd ', 'e*', 'f*', 'g ']
['a ', 'b ', 'c*', 'd ', 'e ', 'f*', 'g ']
['a ', 'b*', 'c ', 'd ', 'e*', 'f ', 'g ']
['a ', 'b ', 'c*', 'd ', 'e*', 'f ', 'g ']
['a ', 'b*', 'c ', 'd ', 'e ', 'f*', 'g ']
['a ', 'b ', 'c*', 'd ', 'e*', 'f*', 'g ']
['a ', 'b*', 'c ', 'd ', 'e ', 'f ', 'g ']