Develop Reference

How to eliminate null valued cells from a CSV dataset using Python?

Each row represents a person(315 total) and each column represent a choice scenario(16 total). Each person responded randomly to 4 consecutive choice scenarios. I want to have four columns consecutively having the responses of each person and do away with any blank cell.Image of the excel sheet
import pandas as pd
df = pd.read_csv(r"C:\Users\Admin\Desktop\Book2.csv")
for (r,c) in df.iterrows():
if df.iat[r,c] is not None:
for i in range(4):
print(str(df.iat[r,c+i]))
UPDATE
I have managed to get data row wise into a list and grouped them in groups of 4 (as i need it). Now how do I keep the elements with values other than ' '?
import csv
rowdata = []
with open(r'C:\Users\ARPLAB31\Desktop\SPdata.csv') as inputfile:
reader = csv.reader(inputfile)
rowdata = list(reader)
r= []
for i in range(1,718,1):
for j in range(28):
if len(rowdata[i][j])!=0:
r.append(rowdata[i][j])
cardref = [r[x:x+4] for x in range(0, len(r),4)] '''cardref contains the partitioned data.'''
print(cardref)
OUTPUT:
[[' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['BB', 'BB', 'CC', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['CC', 'BB', 'CC', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['CC', 'CC', 'CC', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['CC', 'CC', 'AA', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], ['CC', 'BB', 'CC', 'CC'], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' '], [' ', ' ', ' ', ' ']]

Use df.isnull() [ where df is pandas dataframe]
A good resource to find the null value in dataframe in pandas.
https://dzone.com/articles/pandas-find-rows-where-columnfield-is-null

You can read each row, obtain the not null fields, and create a new CSV from there.
For example:
data = ",,2,2,2,2,,,"
arr = filter(None, data.split(",")) #removes null fields
",".join(arr) #"2,2,2,2"

Thanks all of you. I happened to solve my problem with the help of all the above comments. Please mention any changes to the code in the comment.
import pandas
import csv
rowdata = []
''' READING CSV INTO LIST'''
with open('FILE.csv') as inputfile:
reader = csv.reader(inputfile)
rowdata = list(reader)
'''RECORDING THE POSITION OF NON-EMPTY ELEMENTS'''
r= []
for i in range(1,718,1):
for j in range(28):
if len(rowdata[i][j])!=0:
r.append(j)
continue
''' RE-GROUPING LIST AS LIST IN LIST'''
resp_index = [r[x:x+4] for x in range(0, len(r),4)]
print(resp_index)
print(len(resp_index))
'''ELIMINATING BLANK SPACES AND STORING INTO NEW LIST'''
s= []
for i in range(1,718,1):
for j in range(28):
if len(rowdata[i][j])!=0:
s.append(rowdata[i][j])
''' RE-GROUPING LIST AS LIST IN LIST'''
resp_main = [s[x:x+4] for x in range(0, len(s),4)]
print(resp_main)
print(len(resp_main))
pd = pandas.DataFrame(resp_index)
pe = pandas.DataFrame(resp_main)
'''SAVING TO CSV FILES'''
pd.to_csv('INDEX.csv')
pe.to_csv('RESPONSE.csv')

How to get access to every element of the 2d array and than replace them?

arr = [['.' for i in range(4)] for j in range(4)]
for line, i in enumerate(arr):
for column, j in enumerate(i):
print(j, 'at column', column+1, 'line', line+1) # we can know which
# postition takes
# every element
How out of loop to check if coordinate is different with another coordinate.
What I want to get in final:
pseudocode:
#arr[x][y]
arr[1][0] = 'new'
if arr[1][4] - arr[1][0] == 4: # i.e. coord are different by `y` on 4 pos
arr[1][4] = 'new'`
# Before || After
[[' ', 'new', ' ', ' '],|| [[' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ']] || [' ', 'new', ' ', ' ']]
OR
#arr[x][y]
arr[0][0] = 'new'
if arr[3][0] - arr[0][0] == 3: # i.e. coord are different by `x` on 3 pos
arr[3][0] = 'new'`
# Before || After
[['new', ' ', ' ', ' '],|| [[' ', ' ', ' ', 'new'],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' '], || [' ', ' ', ' ', ' '],
[' ', ' ', ' ', ' ']] || [' ', ' ', ' ', ' ']]
Necessarily need to know which list in main list takes position, but how it does outside of loop without numpy, using native python?

Question: How out of loop to check if coordinate is different with another coordinate.
Define the coordinate's using tuple, then compare the tuples:
RC = (1,4)
RC2 = (1,0)
if RC == RC2:
print('Equal')
else:
print('Different')
Type list is 0 Based
You have a list of lists.
RC Coordinate System, the same as with Excel
R == index of list in list == y == Row
C == index inside a list == x == Column
A B C D
C => 0 1 2 3
----------------------
R:0 | [0.0, 0.1, 0.2, 0.3]
R:1 | [1.0, 1.1, 1.2, 1.3]
R:2 | [2.0, 2.1, 2.2, 2.3]
R:3 | [3.0, 3.1, 3.2, 3.3]
Use the opposite
#arr[y][x]
The first index=y is the index of a list in arr, alias row.
The second index=x is the index inside the list selected with y, alias column.

Pythonic way of creating list of lists from list

I would like to create a list of lists from a list.
The list looks like this:
level = [' WWWWWWWWWWWWWWWWW', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C E']
I need to create this:
[[' ', ' ', ' ', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', 'W', ' ', 'C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'W'], ['C', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'E']]
I have done it this way:
listofLists = []
for row in level:
liss = []
for col in row:
liss.append(col)
listofLists.append(liss)
What is a more pythonic way or shorter way of doing this?

>>> listofLists = map(list,level)
(in python3, if you really need a list, do list(map(list, level)))

When you call list() on a string this will return the list of all its characters (including spaces).
level = [' WWWWWWWWWWWWWWWWW', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C W C W', 'C E']
trasnsformed = [list(x) for x in level]

Unique combinations of two lists – comparison at each index position

I've searched pretty thoroughly, and haven't found any info – hopefully I didn't miss anything. I have two lists:
list1 = (a, b, c, d)
list2 = (a, b*, c*, d)
I would like to generate all possible unique 2 list pairings that only look for differences at each index value. For example, the results here would be:
list1_new = (a, b*, c, d)
list2_new = (a, b, c*, d)
Note: I don't care to distinguish between list1 and list2, i.e., list1_new = (a, b*, c*, d) would not be considered unique as it matches the original list2.
I've played around with itertools but haven't been able to figure out how to compare at each index position.
I used small lists for this example, but I actually will have larger lists of 10+ items.

from itertools import product
list1 = ["a ", "b ", "c ", "d ", "e ", "f ", "g "]
list2 = ["a ", "b*", "c*", "d ", "e*", "f*", "g "]
# figure out which offsets have alternable values
crosses = [a != b for a,b in zip(list1, list2)]
offsets = [i for i,cross in enumerate(crosses) if cross]
# decide which offsets will be swapped
basis = [[0,1] for ofs in offsets]
basis[0] = [0] # only the first half - remainder are mirrors
swaps = product(*basis)
next(swaps, None) # skip base state == (list1, list2)
# print all viable swaps
list3 = list1[:]
list4 = list2[:]
for sw in swaps:
# build output lists
for which,offs in zip(sw, offsets):
list3[offs] = [list1, list2][which][offs]
list4[offs] = [list2, list1][which][offs]
# display output
print("\n{}\n{}".format(list3, list4))
gives
['a ', 'b ', 'c ', 'd ', 'e ', 'f*', 'g ']
['a ', 'b*', 'c*', 'd ', 'e*', 'f ', 'g ']
['a ', 'b ', 'c ', 'd ', 'e*', 'f ', 'g ']
['a ', 'b*', 'c*', 'd ', 'e ', 'f*', 'g ']
['a ', 'b ', 'c ', 'd ', 'e*', 'f*', 'g ']
['a ', 'b*', 'c*', 'd ', 'e ', 'f ', 'g ']
['a ', 'b ', 'c*', 'd ', 'e ', 'f ', 'g ']
['a ', 'b*', 'c ', 'd ', 'e*', 'f*', 'g ']
['a ', 'b ', 'c*', 'd ', 'e ', 'f*', 'g ']
['a ', 'b*', 'c ', 'd ', 'e*', 'f ', 'g ']
['a ', 'b ', 'c*', 'd ', 'e*', 'f ', 'g ']
['a ', 'b*', 'c ', 'd ', 'e ', 'f*', 'g ']
['a ', 'b ', 'c*', 'd ', 'e*', 'f*', 'g ']
['a ', 'b*', 'c ', 'd ', 'e ', 'f ', 'g ']

Transposing a 3D list in Python

I have to transpose a 3d list the following way:
Input:
matrix7 = [[['A ', 'E ', 'C#'], ['B ', 'E ', 'C#'], ['C ', 'E ', 'C#']],
[[' ', 'F#', 'D '], [' ', 'F#', 'D '], [' ', 'F#', 'D ']],
[[' ', 'E ', 'B '], [' ', 'E ', 'B '], [' ', 'E ', 'B ']],
[[' ', 'E ', 'C#'], [' ', 'E ', 'C#'], [' ', 'E ', 'C#']],
[[' ', 'F#', 'D '], [' ', 'F#', 'D '], [' ', 'F#', 'D ']],
[[' ', 'E ', 'B '], [' ', 'E ', 'B '], [' ', 'E ', 'B ']],
[[' ', ' ', ' '], [' ', ' ', ' '], [' ', ' ', ' ']],
[[' ', 'E ', 'C#'], [' ', 'E ', 'C#'], [' ', 'E ', 'C#']]]
desired output:
[[['A ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']],
[['B ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'],
[' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']],
[['C ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'],
[' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']]]
I have the following program:
matrix8 = []
for index in matrix7:
matrix8 = numpy.array(matrix7).transpose()
matrix9 = matrix8.tolist()
print matrix9
which is giving me the wrong output:
[[['A ', 'E ', 'C#'], ['B ', 'E ', 'C#'], ['C ', 'E ', 'C#']],
[[' ', 'F#', 'D '], [' ', 'F#', 'D '], [' ', 'F#', 'D ']],
[[' ', 'E ', 'B '], [' ', 'E ', 'B '], [' ', 'E ', 'B ']],
[[' ', 'E ', 'C#'], [' ', 'E ', 'C#'], [' ', 'E ', 'C#']],
[[' ', 'F#', 'D '], [' ', 'F#', 'D '], [' ', 'F#', 'D ']],
[[' ', 'E ', 'B '], [' ', 'E ', 'B '], [' ', 'E ', 'B ']],
[[' ', ' ', ' '], [' ', ' ', ' '], [' ', ' ', ' ']],
[[' ', 'E ', 'C#'], [' ', 'E ', 'C#'], [' ', 'E ', 'C#']]]
Can anyone help me with this?

I think this is what you want:
numpy.transpose(matrix7, axes=(1, 0, 2)).tolist() # The 'axes' attribute tells transpose to swaps axes 0 and 1, leaving the last one alone.
OUTPUT:
[[['A ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']],
[['B ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']],
[['C ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']]]

You don't actually need numpy for this:
>>> [list(x) for x in zip(*matrix7)]
[[['A ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']],
[['B ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']],
[['C ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']]]
Or, if you don't mind getting a list of tuples of lists, just:
>>> list(zip(*matrix7))
[(['A ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']),
(['B ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']),
(['C ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#'])]
Or, in Python 2.x, even less:
>>> zip(*matrix7)
[(['A ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']),
(['B ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#']),
(['C ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', 'E ', 'C#'], [' ', 'F#', 'D '], [' ', 'E ', 'B '], [' ', ' ', ' '], [' ', 'E ', 'C#'])]