I have a list of DNA sequences as given below. I would like to get the consensus sequence which is the highest frequency obtain in that particular position.
test.txt
>human
ATCAATTGCT
>human
GCTAGCTAGC
>human
GCTAGCTAGC
>human
GCTGATCGGC
>human
GCTTACAACG
Using the code below, I obtain the total A, C, G, and T from each position.
Code
from Bio import motifs
output=open("test_output.txt","a")
with open("test.txt") as handle:
motif = motifs.read(handle, 'sites')
output.write(str(motif.counts))
Example output
0 1 2 3 4 5 6 7 8 9
A: 1.00 0.00 0.00 3.00 3.00 0.00 1.00 3.00 0.00 0.00
C: 0.00 4.00 1.00 0.00 0.00 3.00 1.00 0.00 2.00 3.00
G: 4.00 0.00 0.00 1.00 2.00 0.00 0.00 2.00 3.00 1.00
T: 0.00 1.00 4.00 1.00 0.00 2.00 3.00 0.00 0.00 1.00
How can I get the output of each bases stated in the last column?
Desired output
0 1 2 3 4 5 6 7 8 9
A: 1.00 0.00 0.00 3.00 3.00 0.00 1.00 3.00 0.00 0.00
C: 0.00 4.00 1.00 0.00 0.00 3.00 1.00 0.00 2.00 3.00
G: 4.00 0.00 0.00 1.00 2.00 0.00 0.00 2.00 3.00 1.00
T: 0.00 1.00 4.00 1.00 0.00 2.00 3.00 0.00 0.00 1.00
G C T A A C T A G C
Motifs has a consensus method that does exactly what you want:
output.write("\t".join(list(motif.consensus)))
Related
I am pretty new to Python so a few problems occurred
I have an Excel Sheet with different entries and my goal is it to read each entry and automatically assign it to its name. By now this is a simplified sheet and more values could be added so i did not wanted to address each value after another.
So far i did this
import pandas as pd
import numpy as np
df = pd.read_excel('koef.xlsx')
data_array = np.array(df)
XCoeff = []
YCoeff = []
NCoeff = []
VarName = []
for i in range(len(data_array)):
XCoeff.append(data_array[i][1])
XCoeff.pop(0)
for i in range(len(data_array)):
YCoeff.append(data_array[i][2])
YCoeff.pop(0)
for i in range(len(data_array)):
NCoeff.append(data_array[i][3])
NCoeff.pop(0)
for i in range(len(data_array)):
VarName.append(data_array[i][0])
VarName.pop(0)
s1 = "X"
s2 = "Y"
s3 = "N"
XName = [s1 + x for x in VarName]
YName = [s2 + x for x in VarName]
NName = [s3 + x for x in VarName]
In the end i want a list of Variables for X,Y and N where for example the first entries of X would be: Xdel = 0.00, Xdel2 = 4.44, Xdel3 = -2.06 and so on. With these variables i need to do calculations.
The Excel Sheet:
Motion X Y N
0 zero 0.00 0 0.00
1 del 0.00 4.44 -2.06
2 del2 -2.09 -0.24 0.16
3 del3 0.00 -2.95 1.38
4 u -2.20 0 0.00
5 uu 1.50 X 0.00
6 uuu 0.00 0 0.00
7 udot -1.47 0 0.00
8 v 0.11 -24.1 -7.94
9 vv 2.74 2.23 -1.15
10 vvv 0.00 -74.7 2.79
11 vdot 0.00 -16.4 -0.47
12 r -0.07 4.24 -3.32
13 rr 0.58 0.56 -0.27
14 rrr 0.00 2.58 -1.25
15 rdot 0.00 -0.46 -0.75
16 vr 13.10 0 0.00
17 vrr 0.00 -40.3 8.08
18 vvr 0.00 -9.9 -3.37
19 udel 0.00 -4.56 2.32
20 vdel2 0.00 5.15 -1.17
21 vvdel 0.00 7.4 -3.41
22 rdel2 0.00 -0.51 -0.58
23 rrdel 0.00 -0.98 0.43
I hope the problem is stated clear, if not feel free to ask.
Thank You
So far i got the lists at least working but i struggle to merge them.
If you load the Excel sheet in a certain manner, accessing the cells by name can be simple. I think this will give you the cell access you need:
import pandas as pd
# Read the Excel sheet, index by the Motion column.
df = pd.read_excel('koef.xlsx', index_col='Motion')
print(df)
print(df.Y.del3) # specific cells
print(df.N.vvr)
Output:
X Y N
Motion
zero 0.00 0 0.00
del 0.00 4.44 -2.06
del2 -2.09 -0.24 0.16
del3 0.00 -2.95 1.38
u -2.20 0 0.00
uu 1.50 X 0.00
uuu 0.00 0 0.00
udot -1.47 0 0.00
v 0.11 -24.1 -7.94
vv 2.74 2.23 -1.15
vvv 0.00 -74.7 2.79
vdot 0.00 -16.4 -0.47
r -0.07 4.24 -3.32
rr 0.58 0.56 -0.27
rrr 0.00 2.58 -1.25
rdot 0.00 -0.46 -0.75
vr 13.10 0 0.00
vrr 0.00 -40.3 8.08
vvr 0.00 -9.9 -3.37
udel 0.00 -4.56 2.32
vdel2 0.00 5.15 -1.17
vvdel 0.00 7.4 -3.41
rdel2 0.00 -0.51 -0.58
rrdel 0.00 -0.98 0.43
-2.95
-3.37
Caveat: The column/row names need to also be valid Python identifiers, but if not you can use for example df['Y']['del3'] syntax as well. Using valid Python identifiers makes the syntax easier to type.
I am trying to fill the dataframe with certain condition but I can not find the appropriate solution. I have a bit larger dataframe bet let's say that my pandas dataframe looks like this:
0
1
2
3
4
5
0.32
0.40
0.60
1.20
3.40
0.00
0.17
0.12
0.00
1.30
2.42
0.00
0.31
0.90
0.80
1.24
4.35
0.00
0.39
0.00
0.90
1.50
1.40
0.00
And I want to update the values, so that if 0.00 appears once in a row (row 2 and 4) that until the end all the values are 0.00. Something like this:
0
1
2
3
4
5
0.32
0.40
0.60
1.20
3.40
0.00
0.17
0.12
0.00
0.00
0.00
0.00
0.31
0.90
0.80
1.24
4.35
0.00
0.39
0.00
0.00
0.00
0.00
0.00
I have tried with
for t in range (1,T-1):
data= np.where(df[t-1]==0,0,df[t])
and several others ways but I couldn't get what I want.
Thanks!
Try as follows:
Select from df with df.eq(0). This will get us all zeros and the rest as NaN values.
Now, add df.ffill along axis=1. This will continue all the zeros through to the end of each row.
Finally, change the dtype to bool by chaining df.astype, thus turning all zeros into False, and all NaN values into True.
We feed the result to df.where. For all True values, we'll pick from the df itself, for all False values, we'll insert 0.
df = df.where(df[df.eq(0)].ffill(axis=1).astype(bool), 0)
print(df)
0 1 2 3 4 5
0 0.32 0.40 0.6 1.20 3.40 0.0
1 0.17 0.12 0.0 0.00 0.00 0.0
2 0.31 0.90 0.8 1.24 4.35 0.0
3 0.39 0.00 0.0 0.00 0.00 0.0
The value of column A can be -1, 0 and 1. How to create a new column with conditions: column A; value different from 0 in the line below (-1,1), add the value of column X, if value of column X equals 0 then add the next value different from 0. In the other lines of this new column, assign the value 0.
import pandas as pd
d = {'a': [-1,-1,-1,0,0,-1,-1,-1,-1,-1,-1,-1,-1,0,0,0,0,0,1,1,1,0,0,0,0,1,1,1,1,1],
'x': [0.00,-2.13,0.00,0.00,0.00,0.00,0.00,0.21,-0.63,0.00,0.29,-0.11,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.62,0.00,
-0.36,0.00,-0.03,0.00,0.00,0.00,0.22,0.05,0.00]}
df = pd.DataFrame(data=d)
df
desired result
a x r
0 -1 0.00 0.00
1 -1 -2.13 0.00
2 -1 0.00 0.00
3 0 0.00 0.00
4 0 0.00 0.00
5 -1 0.00 0.00
6 -1 0.00 0.00
7 -1 0.21 0.21
8 -1 -0.63 0.00
9 -1 0.00 0.00
10 -1 0.29 0.00
11 -1 -0.11 0.00
12 -1 0.00 0.00
13 0 0.00 0.00
14 0 0.00 0.00
15 0 0.00 0.00
16 0 0.00 0.00
17 0 0.00 0.00
18 1 0.00 0.00
19 1 0.62 0.62
20 1 0.00 0.00
21 0 -0.36 0.00
22 0 0.00 0.00
23 0 -0.03 0.00
24 0 0.00 0.00
25 1 0.00 0.00
26 1 0.00 0.00
27 1 0.22 0.22
28 1 0.05 0.00
29 1 0.00 0.00
I've been stuck with an engineering problem thats Python/Pandas related. I'd appreciate any help given.
I've simplified the numbers so I can better explain myself.
I have something similar to the following:
positioning(x-axis)
Calculated difference
1
0.25
0.05
2
0.75
0.06
3
1.25
0.02
4
0.25
0.05
5
0.75
0.05
6
1.25
0.02
7
0.25
0.09
8
0.75
0.01
9
1.25
0.02
10
0.25
0.05
What I need to do is re-organise the calculated difference based on the x-axis positioning.
So it looks something like this:
(0.25)
(0.75)
(1.25)
0.05
0
0
0
0.06
0
0
0
0.02
0.5
0
0
0
0.5
0
0
0
0.02
0.09
0
0
0
0.01
0
0
0
0.02
0.05
0
0
As you can see, I need to organize everything based on the x-positioning.
What is the best approach to this problem? Keep in mind I have 2000+ rows and the x positioning is dynamic but I'm currently working till up to 50(so a lot of columns).
I hope I've clarified the question.
Use pd.get_dummies:
In [10]: pd.get_dummies(df['positioning(x-axis)']).mul(df['Calculated difference'],axis=0)
Out[10]:
0.25 0.75 1.25
1 0.05 0.00 0.00
2 0.00 0.06 0.00
3 0.00 0.00 0.02
4 0.05 0.00 0.00
5 0.00 0.05 0.00
6 0.00 0.00 0.02
7 0.09 0.00 0.00
8 0.00 0.01 0.00
9 0.00 0.00 0.02
10 0.05 0.00 0.00
Just do pivot
df.pivot(columns='positioning(x-axis)',values='Calculated difference').fillna(0)
Out[363]:
Calculated 0.25 0.75 1.25
0 0.05 0.00 0.00
1 0.00 0.06 0.00
2 0.00 0.00 0.02
3 0.05 0.00 0.00
4 0.00 0.05 0.00
5 0.00 0.00 0.02
6 0.09 0.00 0.00
7 0.00 0.01 0.00
8 0.00 0.00 0.02
9 0.05 0.00 0.00
factorize
i, p = pd.factorize(df['positioning(x-axis)'])
d = df['Calculated difference'].to_numpy()
a = np.zeros_like(d, shape=(len(df), len(p)))
a[np.arange(len(df)), i] = d
pd.DataFrame(a, df.index, p)
0.25 0.75 1.25
0 0.05 0.00 0.00
1 0.00 0.06 0.00
2 0.00 0.00 0.02
3 0.05 0.00 0.00
4 0.00 0.05 0.00
5 0.00 0.00 0.02
6 0.09 0.00 0.00
7 0.00 0.01 0.00
8 0.00 0.00 0.02
9 0.05 0.00 0.00
One way to do this would be to use pandas' pivot and then to reset the index.
Given a data frame like this:
positioning(x-axis) Calculated difference
0 0.0 0.61
1 0.0 0.96
2 0.0 0.56
3 0.0 0.91
4 0.0 0.57
5 0.0 0.67
6 0.1 0.71
7 0.1 0.71
8 0.1 0.95
9 0.1 0.89
10 0.1 0.61
df.pivot(columns='positioning(x-axis)', values='Calculated difference').reset_index().drop(columns=['index']).fillna(0)
positioning(x-axis) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
0 0.61 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
1 0.96 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.22 0.00 0.00
3 0.00 0.66 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.13 0.00 0.00
5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
6 0.00 0.00 0.00 0.91 0.00 0.00 0.00 0.00 0.00 0.00 0.00
7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.85
8 0.00 0.00 0.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
9 0.00 0.91 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
I have a process that loops through two lists, one being relatively large while the other being significantly smaller.
Example:
larger_list = list(range(15000))
smaller_list = list(range(2500))
for ll in larger_list:
for sl in smaller_list:
pass
I scaled the sized down of the lists to test performance, and I noticed there is a decent difference between which list is looped through first.
import timeit
larger_list = list(range(150))
smaller_list = list(range(25))
def large_then_small():
for ll in larger_list:
for sl in smaller_list:
pass
def small_then_large():
for sl in smaller_list:
for ll in larger_list:
pass
print('Larger -> Smaller: {}'.format(timeit.timeit(large_then_small)))
print('Smaller -> Larger: {}'.format(timeit.timeit(small_then_large)))
>>> Larger -> Smaller: 114.884992572
>>> Smaller -> Larger: 98.7751009799
At first glance, they look identical - however there is 16 second difference between the two functions.
Why is that?
When you disassemble one of your functions you get:
>>> dis.dis(small_then_large)
2 0 SETUP_LOOP 31 (to 34)
3 LOAD_GLOBAL 0 (smaller_list)
6 GET_ITER
>> 7 FOR_ITER 23 (to 33)
10 STORE_FAST 0 (sl)
3 13 SETUP_LOOP 14 (to 30)
16 LOAD_GLOBAL 1 (larger_list)
19 GET_ITER
>> 20 FOR_ITER 6 (to 29)
23 STORE_FAST 1 (ll)
4 26 JUMP_ABSOLUTE 20
>> 29 POP_BLOCK
>> 30 JUMP_ABSOLUTE 7
>> 33 POP_BLOCK
>> 34 LOAD_CONST 0 (None)
37 RETURN_VALUE
>>>
Looking at address 29 & 30, it looks like these will execute every time the inner loop ends. The two loops look basically the same, but these two instructions are executed each time the inner loop exits. Having the smaller number on the inside would cause these to be executed more often, hence increasing the time (vs the larger number on the inner loop).
This same phenomenon was under discussion in this duplicate and got me interested in what goes on in the C land of CPython. Built python with:
% ./configure --enable-profiling
% make coverage
Tests
% ./python -c "larger_list = list(range(15000))
smaller_list = list(range(2500))
for sl in smaller_list:
for ll in larger_list:
pass"
% mv gmon.out soflgmon.out
% ./python -c "larger_list = list(range(15000))
smaller_list = list(range(2500))
for ll in larger_list:
for sl in smaller_list:
pass"
% mv gmon.out lofsgmon.out
Results
Short list of long lists (total time for a single run 1.60):
% gprof python soflgmon.out|head -n40
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
46.25 0.74 0.74 3346 0.00 0.00 PyEval_EvalFrameEx
25.62 1.15 0.41 37518735 0.00 0.00 insertdict
14.38 1.38 0.23 37555121 0.00 0.00 lookdict_unicode_nodummy
7.81 1.50 0.12 37506675 0.00 0.00 listiter_next
4.06 1.57 0.07 37516233 0.00 0.00 PyDict_SetItem
0.62 1.58 0.01 2095 0.00 0.00 _PyEval_EvalCodeWithName
0.62 1.59 0.01 3 0.00 0.00 untrack_dicts
0.31 1.59 0.01 _PyDict_SetItem_KnownHash
0.31 1.60 0.01 listiter_len
0.00 1.60 0.00 87268 0.00 0.00 visit_decref
0.00 1.60 0.00 73592 0.00 0.00 visit_reachable
0.00 1.60 0.00 71261 0.00 0.00 _PyThreadState_UncheckedGet
0.00 1.60 0.00 49742 0.00 0.00 _PyObject_Alloc
0.00 1.60 0.00 48922 0.00 0.00 PyObject_Malloc
0.00 1.60 0.00 48922 0.00 0.00 _PyObject_Malloc
0.00 1.60 0.00 47487 0.00 0.00 PyDict_GetItem
0.00 1.60 0.00 44246 0.00 0.00 _PyObject_Free
0.00 1.60 0.00 43637 0.00 0.00 PyObject_Free
0.00 1.60 0.00 30034 0.00 0.00 slotptr
0.00 1.60 0.00 24892 0.00 0.00 type_is_gc
0.00 1.60 0.00 24170 0.00 0.00 r_byte
0.00 1.60 0.00 23774 0.00 0.00 PyErr_Occurred
0.00 1.60 0.00 20371 0.00 0.00 _PyType_Lookup
0.00 1.60 0.00 19930 0.00 0.00 PyLong_FromLong
0.00 1.60 0.00 19758 0.00 0.00 r_string
0.00 1.60 0.00 19080 0.00 0.00 _PyLong_New
0.00 1.60 0.00 18887 0.00 0.00 lookdict_unicode
0.00 1.60 0.00 18878 0.00 0.00 long_dealloc
0.00 1.60 0.00 17639 0.00 0.00 PyUnicode_InternInPlace
0.00 1.60 0.00 17502 0.00 0.00 rangeiter_next
0.00 1.60 0.00 14776 0.00 0.00 PyObject_GC_UnTrack
0.00 1.60 0.00 14578 0.00 0.00 descr_traverse
0.00 1.60 0.00 13520 0.00 0.00 r_long
0.00 1.60 0.00 13058 0.00 0.00 PyUnicode_New
0.00 1.60 0.00 12298 0.00 0.00 _Py_CheckFunctionResult
...
Long list of short lists (total time for a single run 1.64):
gprof python lofsgmon.out|head -n40
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls s/call s/call name
48.78 0.80 0.80 3346 0.00 0.00 PyEval_EvalFrameEx
17.99 1.09 0.29 37531168 0.00 0.00 insertdict
11.59 1.28 0.19 37531675 0.00 0.00 listiter_next
11.28 1.47 0.18 37580156 0.00 0.00 lookdict_unicode_nodummy
6.71 1.58 0.11 37528666 0.00 0.00 PyDict_SetItem
1.22 1.60 0.02 _PyDict_SetItem_KnownHash
0.61 1.61 0.01 5525 0.00 0.00 update_one_slot
0.61 1.62 0.01 120 0.00 0.00 PyDict_Merge
0.30 1.62 0.01 18178 0.00 0.00 lookdict_unicode
0.30 1.63 0.01 11988 0.00 0.00 insertdict_clean
0.30 1.64 0.01 listiter_len
0.30 1.64 0.01 listiter_traverse
0.00 1.64 0.00 96089 0.00 0.00 _PyThreadState_UncheckedGet
0.00 1.64 0.00 87245 0.00 0.00 visit_decref
0.00 1.64 0.00 74743 0.00 0.00 visit_reachable
0.00 1.64 0.00 62232 0.00 0.00 _PyObject_Alloc
0.00 1.64 0.00 61412 0.00 0.00 PyObject_Malloc
0.00 1.64 0.00 61412 0.00 0.00 _PyObject_Malloc
0.00 1.64 0.00 59815 0.00 0.00 PyDict_GetItem
0.00 1.64 0.00 55231 0.00 0.00 _PyObject_Free
0.00 1.64 0.00 54622 0.00 0.00 PyObject_Free
0.00 1.64 0.00 36274 0.00 0.00 PyErr_Occurred
0.00 1.64 0.00 30034 0.00 0.00 slotptr
0.00 1.64 0.00 24929 0.00 0.00 type_is_gc
0.00 1.64 0.00 24617 0.00 0.00 _PyObject_GC_Alloc
0.00 1.64 0.00 24617 0.00 0.00 _PyObject_GC_Malloc
0.00 1.64 0.00 24170 0.00 0.00 r_byte
0.00 1.64 0.00 20958 0.00 0.00 PyObject_GC_Del
0.00 1.64 0.00 20371 0.00 0.00 _PyType_Lookup
0.00 1.64 0.00 19918 0.00 0.00 PyLong_FromLong
0.00 1.64 0.00 19758 0.00 0.00 r_string
0.00 1.64 0.00 19068 0.00 0.00 _PyLong_New
0.00 1.64 0.00 18845 0.00 0.00 long_dealloc
0.00 1.64 0.00 18507 0.00 0.00 _PyObject_GC_New
0.00 1.64 0.00 17639 0.00 0.00 PyUnicode_InternInPlace
...
The difference is marginal (2.4%), and profiling adds to run time, so it is difficult to say how much it actually would've been. The total time also includes the creation of the test lists, so that hides the true difference further.
The reason for the 16s difference in the original test is that timeit.timeit runs the given statement or function number=1000000 times by default, so that would add up to a whopping 40,000s in this case. Don't quote that value though, as it is an artifact of profiling. With your original test code and non profiling python3 on this machine I get:
Larger -> Smaller: 40.29234626500056
Smaller -> Larger: 33.09413992699956
which would mean a difference of
In [1]: (40.29234626500056-33.09413992699956)/1000000
Out[1]: 7.198206338001e-06
per single run (7.2µs), 18% in total.
So as stated in the former answer, POP_BLOCK gets executed more, but it's not just that, but the whole inner loop setup:
0.00 1.64 0.00 16521 0.00 0.00 PyFrame_BlockSetup
0.00 1.64 0.00 16154 0.00 0.00 PyFrame_BlockPop
Compared to the short list of long lists:
0.00 1.60 0.00 4021 0.00 0.00 PyFrame_BlockSetup
0.00 1.60 0.00 3748 0.00 0.00 set_next
0.00 1.60 0.00 3654 0.00 0.00 PyFrame_BlockPop
That has negligible impact though.