Develop Reference

How to transform a multi dimensional array from a CSV file into a list

screenshot of the csv file
Hi(sorry if this is a dump question)..i have a data set as CSV file ...every row contains 44 column and every cell containes 44 float number separated by two spaces like this(look at the screenshot) ...i tried CSV readline/s + numpy and non of them worked
i want to take every row as a list with[1936] variable (44*44)
and then combine the whole data set into 2d array ...my_data[n_of_samples][1936]

so as stated by user ybl, this is not a CSV. It's not even close to being a CSV.
This means that you have to implement some processing to turn this into something useable. I put the screenshot through an OCR to extract the actual text values, but next time provide the input file. Screenshots of data are annoying to work with.
The processing you need to to is to find the start and end of the rows, using the [ and ] characters respectively. Then you split this data with the basic string.split() which doesn't care about the number of spaces.
Try the code below and see if that works for the input file.
rows = []
current_row = ""
with open("somefile.txt") as infile:
for line in infile.readlines():
cleaned = line.replace('"', '').replace("\n", " ")
if "]" in cleaned:
current_row = f"{current_row} {cleaned.split(']')[0]}"
rows.append(current_row.split())
current_row = ""
cleaned = cleaned.split(']')[1]
if "[" in cleaned:
cleaned = cleaned.split("[")[1]
current_row = f"{current_row} {cleaned}"
for row in rows:
print(len(row))
output
44
44
44
input file:
"[ 1.79619717e+04 1.09988207e+02 4.13270009e+01 1.72227906e+01
1.06178751e+01 5.20957856e+00 7.50891645e+00 4.57943370e+00
2.65572713e+00 2.96725867e-01 2.43040664e+00 1.32822091e+00
4.09853169e-01 1.18412873e+00 6.43398990e-01 1.23796528e+00
9.63975374e-02 2.95295579e-01 7.68998970e-01 4.98040980e-01
2.84036936e-01 1.76004564e-01 1.43527613e-01 1.64765236e-01
1.51171075e-01 1.02586637e-01 3.27835810e-02 1.21872869e-02
-7.59824907e-02 8.48217334e-02 7.29953754e-02 4.89750588e-02
5.89426950e-02 5.05485266e-02 2.34761263e-02 -2.41095452e-02
5.15952510e-02 1.39933210e-02 2.12354074e-02 3.40820680e-03
-2.57466949e-03 -1.06481222e-02 -8.35155410e-03 1.21653512e-12]","[-6.12189619e+02 1.03584744e+04 2.34417495e+02 7.01761526e+01
3.92495170e+01 1.81609738e+01 2.58114624e+01 1.52275550e+01
8.59676934e+00 9.45036161e-01 7.71943506e+00 4.17516432e+00
1.27920413e+00 3.68862368e+00 1.99582544e+00 3.82999035e+00
2.96068511e-01 9.06341796e-01 2.35621065e+00 1.52094079e+00
8.64565916e-01 5.34605108e-01 4.35456793e-01 4.99450615e-01
4.57778770e-01 3.10324997e-01 9.90860520e-02 3.68281889e-02
-2.29532895e-01 2.56108491e-01 2.20284123e-01 1.47727878e-01
1.77724506e-01 1.52350751e-01 7.07318164e-02 -7.26252404e-02
1.55364050e-01 4.21222079e-02 6.39113311e-02 1.02558665e-02
-7.74736016e-03 -3.20368093e-02 -2.51241082e-02 1.21653512e-12]","[-5.03959282e+02 -5.64452044e+02 7.90433958e+03 1.94146598e+02
1.06178751e+01 5.20957856e+00 7.50891645e+00 4.57943370e+00
2.65572713e+00 2.96725867e-01 2.43040664e+00 1.32822091e+00
4.09853169e-01 1.18412873e+00 6.43398990e-01 1.23796528e+00
9.63975374e-02 2.95295579e-01 7.68998970e-01 4.98040980e-01
2.84036936e-01 1.76004564e-01 1.43527613e-01 1.64765236e-01
1.51171075e-01 1.02586637e-01 3.27835810e-02 1.21872869e-02
-7.59824907e-02 8.48217334e-02 7.29953754e-02 4.89750588e-02
5.89426950e-02 5.05485266e-02 2.34761263e-02 -2.41095452e-02
5.15952510e-02 1.39933210e-02 2.12354074e-02 3.40820680e-03
-2.57466949e-03 -1.06481222e-02 -8.35155410e-03 1.21653512e-12]"

The option is this:
import numpy as np
import csv
c = np.array([n_of_samples])
with open('cocacola_sick.csv') as f:
p = csv.reader(f) # read file as csv
for s in p:
a = ','.join(s) # concatenate all lines into one line
a = a.replace("\n", "") # remove line breaks
b = np.array(np.mat(a))
my_data = np.vstack((c,b))
print(my_data)

Extracting lines from multiline string with various variable length sections

I'm working with a pandas dataframe containing a large block of plain text for each row. The block of text has the following format:
Year 1
... (variable # of lines)
7. Stuff
... (variable # of lines, can be 0)
TOTAL Stuff
(single line, numeric)
... (variable # of lines)
Services
(single line)
... (variable # of lines)
Year 2
... (same format as prev)
<repeats for n years>
TOTAL
... (same format as years)
Justification
... (variable number of lines)
<repeat m times>
I'm trying to extract the plain text under the "7. Stuff" and "Justification" headings as well a numerical values for "TOTAL Stuff". My current code creates a list based on the line breaks and iterates through them, but I feel like this is not efficient. It my current implementation also only works when there is one cycle of years -> Total -> Justification (not m).
Here is my parse_text function. Any help on making it more 'pythonic' or just efficient in general is greatly appreciated.
def parse_budget_text(row):
stuff_value = 0
stuff_text = ''
justification_txt = ''
#ensure text is not hidden within a list
text = row['text_raw']
#parse and sum equipment lines
line_iter = iter([line.strip() for line in text.split("\n")])
total_flag = False
justification_flag = False
for line in line_iter:
#find each yearly section
if line.startswith("YEAR"):
while not line.startswith("7. Stuff"):
line = next(line_iter)
line = next(line_iter)
while not line.startswith("Services"):
if ("TOTAL Stuff" not in line) and (not is_number(line)) and (line[0] != "$"):
stuff_txt += line+'; '
line = next(line_iter)
#find total summary
elif line.startswith("TOTAL"):
cumulative_flag = True
while not line.startswith("TOTAL Stuff"):
line =next(line_iter)
stuff_value += int(next(line_iter).replace(',',''))
#find Justification line
elif line.startswith("Justification") and cumulative_flag:
justification_flag = True
#extract justification
elif justification_flag == True:
justification_txt += line
return pd.Series({'raw_text': text, 'Stuff_val': stuff_value, 'Stuff_txt': stuff_txt,})

How to write regular expressions to match white space delimited multi-line column data

So I have log files that come in the form of
n400_108tb_48gb 2 G 1,3-7 1 20G / 286T (< 1% )
n400_108tb_48gb:1 1 D 1-3:bay1-6 - 2.1G / 48T (< 1% )
n400_108tb_48gb:3 3 D 1-3:bay7-12 - 1.9G / 48T (< 1% )
n400_108tb_48gb:4 4 D 1-3:bay13-18 - 10G / 48T (< 1% )
n400_108tb_48gb:5 5 D 1-3:bay19-24 - 2.0G / 48T (< 1% )
n400_108tb_48gb:6 6 D 1-3:bay25-30 - 2.2G / 48T (< 1% )
n400_108tb_48gb:7 7 D 1-3:bay31-36 - 1.7G / 48T (< 1% )
That seems nice and simple to deal with so I can just write regular expressions to deal with that one line at a time.
([0-9a-z_:]*)\s*([1-9])\s*([DGPTE])\s*([0-9a-z_:,-]*)\s*([1-9])\s*([0-9.]+[KMGTPE]).*?([0-9]*[KMGTPE])
I mean, that's ugly but I can simplify it to
_name = r"([0-9a-z_:]*)\s*"
_id = r"([1-9])
_type = r"([DGPTE])"
_members = r"([0-9a-z_:,-]*)"
_vhs = r"([1-9-])"
_used = r"([0-9.]*[KMGTPE])"
_size = r"([0-9.]*[KMGTPE])"
_disk_protections_regex_string = r"{0}\s*{1}\s*{2}\s*{3}\s*{4}\s*{5}.*?{6}".format(
_name,
_id,
_type,
_members,
_vhs,
_used,
_size,)
Then I discovered that I have to parse files with this format.
s200_13tb_400gb 1 +3 system, vhs_de 1:0-23, 1 53T / 218T (25% )
-ssd_48gb-ram ny_writes, vhs 2:0-23, 3:0-
_hide_spare, 1,3-19,21-25
ssd_metadata , 4:0-23,
5:0-23,
6:0-23,
7:0-23,
8:0-23,
9:0-23,
10:0-23,
11:0-23,
12:0-23,
13:0-23,
14:0-23,
15:0-23,
16:0-23,
17:0-23,
18:2-25
and suddenly The expected values are
s200_13tb_400gb-ssd_48gb-ram
system vhs_deny_writes, vhs_hide_spare, ssd_metadata
1:0-23, 2:0-23, 3:0-1,3-19,21-25, 4:0-23, 5:0-23, 6:0-23, 7:0-23, 8:0-23, 9:0-23, 10:0-23, 11:0-23, 12:0-23, 13:0-23, 14:0-23, 15:0-23, 16:0-23, 17:0-23, 18:0-23,
As well as the original formatting I presented. I don't even know where to start with white space delimited column separated values.

I've created a more dynamic method, which finds the column definitions itself.
Explanation
The script first looks in the file for columns where in each
line the character is a whitespace.
It then defines the data column definitions based on being between whitespace columns. + [len(content[0])] adds an additional whitespace column at the end making the last data column accessible if needed.
The data is extracted with the defined columns.
The data is printed if it matches the specific defined patterns. Warning: If you have multiple records per file, you will have to change this step.
Code
import re
from collections import Counter
# Patterns to save in the end, [name, attr, values]
patterns = [r"^([0-9a-z_-]{4,}$)", r"^([a-z_,\s]*$)", r"([0-9:,\s-]{4,})$"]
# Get file content, remove any trailing empty line.
with open('/path/to/my/file') as f:
content = f.read().split('\n')
if not content[-1]:
content = content[:-1]
# 1) Find all single character columns in content with only whitespaces.
no_lines = len(content)
whitespaces = [i for l in content for i, char in enumerate(l) if char == ' ']
whi_columns = [k for k, v in Counter(whitespaces).iteritems() if v == no_lines]
# .items() in python3
# 2) Get all real columns that are between whitespace columns.
columns_defs = []
for i, whi_col in enumerate(whi_columns + [len(content[0])]):
if whi_col and not i: #special first column
columns_defs.append(slice(whi_col))
if whi_col > whi_columns[i - 1] + 1:
columns_defs.append(slice(whi_columns[i - 1] + 1, whi_col))
# 3) Extract columns from file content.
data_columns = [[line[col].strip() for line in content] for col in columns_defs]
# 4) Save columns fitting patterns.
for data_col in data_columns:
data = ''.join(data_col)
if re.match(r'|'.join(patterns), data):
print data
Output
s200_13tb_400gb-ssd_48gb-ram
system, vhs_deny_writes, vhs_hide_spare,ssd_metadata
1:0-23,2:0-23, 3:0-1,3-19,21-25, 4:0-23,5:0-23,6:0-23,7:0-23,8:0-23,9:0-23,10:0-23,11:0-23,12:0-23,13:0-23,14:0-23,15:0-23,16:0-23,17:0-23,18:2-25

Define slices for the columns then aggregate the data in each line
col_1 = slice(17)
col_2 = slice(25,40)
col_3 = slice(41,54)
col_4 = slice(55,None)
one, two, three, four = list(), list(), list(), list()
with open('file.txt') as f:
for line in f:
one.append(line[col_1])
two.append(line[col_2])
three.append(line[col_3])
four.append(line[col_4])
print ''.join(item.strip() for item in one)
print ''.join(item.strip() for item in two)
print ''.join(item.strip() for item in three)
print ''.join(item.strip() for item in four)
>>>
s200_13tb_400gb-ssd_48gb-ram
system, vhs_deny_writes, vhs_hide_spare,ssd_metadata
1:0-23,2:0-23, 3:0-1,3-19,21-25, 4:0-23,5:0-23,6:0-23,7:0-23,8:0-23,9:0-23,10:0-23,11:0-23,12:0-23,13:0-23,14:0-23,15:0-23,16:0-23,17:0-23,18:2-25
53T / 218T (25% )
>>>
This will extract data from the collimated format shown in the example. If there are multiple records in a file, the record delimiter needs to be determined.

How do you make tables with previously stored strings?

So the question basically gives me 19 DNA sequences and wants me to makea basic text table. The first column has to be the sequence ID, the second column the length of the sequence, the third is the number of "A"'s, 4th is "G"'s, 5th is "C", 6th is "T", 7th is %GC, 8th is whether or not it has "TGA" in the sequence. Then I get all these values and write a table to "dna_stats.txt"
Here is my code:
fh = open("dna.fasta","r")
Acount = 0
Ccount = 0
Gcount = 0
Tcount = 0
seq=0
alllines = fh.readlines()
for line in alllines:
if line.startswith(">"):
seq+=1
continue
Acount+=line.count("A")
Ccount+=line.count("C")
Gcount+=line.count("G")
Tcount+=line.count("T")
genomeSize=Acount+Gcount+Ccount+Tcount
percentGC=(Gcount+Ccount)*100.00/genomeSize
print "sequence", seq
print "Length of Sequence",len(line)
print Acount,Ccount,Gcount,Tcount
print "Percent of GC","%.2f"%(percentGC)
if "TGA" in line:
print "Yes"
else:
print "No"
fh2 = open("dna_stats.txt","w")
for line in alllines:
splitlines = line.split()
lenstr=str(len(line))
seqstr = str(seq)
fh2.write(seqstr+"\t"+lenstr+"\n")
I found that you have to convert the variables into strings. I have all of the values calculated correctly when I print them out in the terminal. However, I keep getting only 19 for the first column, when it should go 1,2,3,4,5,etc. to represent all of the sequences. I tried it with the other variables and it just got the total amounts of the whole file. I started trying to make the table but have not finished it.
So my biggest issue is that I don't know how to get the values for the variables for each specific line.
I am new to python and programming in general so any tips or tricks or anything at all will really help.
I am using python version 2.7

Well, your biggest issue:
for line in alllines: #1
...
fh2 = open("dna_stats.txt","w")
for line in alllines: #2
....
Indentation matters. This says "for every line (#1), open a file and then loop over every line again(#2)..."
De-indent those things.

This puts the info in a dictionary as you go and allows for DNA sequences to go over multiple lines
from __future__ import division # ensure things like 1/2 is 0.5 rather than 0
from collections import defaultdict
fh = open("dna.fasta","r")
alllines = fh.readlines()
fh2 = open("dna_stats.txt","w")
seq=0
data = dict()
for line in alllines:
if line.startswith(">"):
seq+=1
data[seq]=defaultdict(int) #default value will be zero if key is not present hence we can do +=1 without originally initializing to zero
data[seq]['seq']=seq
previous_line_end = "" #TGA might be split accross line
continue
data[seq]['Acount']+=line.count("A")
data[seq]['Ccount']+=line.count("C")
data[seq]['Gcount']+=line.count("G")
data[seq]['Tcount']+=line.count("T")
data[seq]['genomeSize']+=data[seq]['Acount']+data[seq]['Gcount']+data[seq]['Ccount']+data[seq]['Tcount']
line_over = previous_line_end + line[:3]
data[seq]['hasTGA']= data[seq]['hasTGA'] or ("TGA" in line) or (TGA in line_over)
previous_line_end = str.strip(line[-4:]) #save previous_line_end for next line removing new line character.
for seq in data.keys():
data[seq]['percentGC']=(data[seq]['Gcount']+data[seq]['Ccount'])*100.00/data[seq]['genomeSize']
s = '%(seq)d, %(genomeSize)d, %(Acount)d, %(Ccount)d, %(Tcount)d, %(Tcount)d, %(percentGC).2f, %(hasTGA)s'
fh2.write(s % data[seq])
fh.close()
fh2.close()

How do I convert integers into high-resolution times in Python? Or how do I keep Python from dropping zeros?

Currently, I'm using this to calculate the time between two messages and listing the times if they are above 20 seconds.
def time_deltas(infile):
entries = (line.split() for line in open(INFILE, "r"))
ts = {}
for e in entries:
if " ".join(e[2:5]) == "OuchMsg out: [O]":
ts[e[8]] = e[0]
elif " ".join(e[2:5]) == "OuchMsg in: [A]":
in_ts, ref_id = e[0], e[7]
out_ts = ts.pop(ref_id, None)
yield (float(out_ts),ref_id[1:-1],(float(in_ts)*10000 - float(out_ts)*10000))
n = (float(in_ts)*10000 - float(out_ts)*10000)
if n> 20:
print float(out_ts),ref_id[1:-1], n
INFILE = 'C:/Users/klee/Documents/text.txt'
import csv
with open('output_file1.csv', 'w') as f:
csv.writer(f).writerows(time_deltas(INFILE))
However, there are two major errors. First of all, python drops zeros when the time is before 10, ie. 0900. And, it drops zeros making the time difference not accurate.
It looks like:
130203.08766
when it should be:
130203.087660

You are yielding floats, so the csv writer turns those floats into strings as it pleases.
If you want your output values to be a certain format, yield a string in that format.
Perhaps something like this?
print "%04.0f" % (900) # prints 0900