Develop Reference

Retrieve many nested statements at the same time for a single object with lxml Python

I am working with big xml where I am retrieving many different properties, and now I am trying to retrieve comment category property and connect it to the text between the tags. However, there are 3 different situations that I need to handle. XML example:
<comment-list>
<comment category="Derived from sampling site"> Peripheral blood </comment>
<comment category="Transformant">
<cv-term terminology="NCBI-Taxonomy" accession="10376">Epstein-Barr virus (EBV)</cv-term>
</comment>
<comment category="Sequence variation"> Hemizygous for FMR1 >200 CGG repeats (PubMed=25776194)
</comment>
<comment category="Monoclonal antibody target">
<xref-list>
<xref database="UniProtKB" category="Sequence databases" accession="Q5T5X7">
<property-list>
<property name="gene/protein designation" value="Human BEND3"/>
</property-list>
<url><![CDATA[https://www.uniprot.org/uniprot/Q5T5X7]]></url>
</xref>
</xref-list>
</comment>
</comment-list>
When <comment> does not have child tags under. Then I need to retrieve comment category property
and connect it with the text between the tags.
When <comment> has a <cv-term> tag nested underneath. Then I need to retrieve comment category,
cv-term terminology, cv-term accession and the text between the cv-term tags.
When <comment> has several tags nested underneath: <xref-list>-<xref>-<property-list>-
<property>-<url>. In this case I need to retrieve: comment category,
xref database property, xref accession property, and property value property.
I am using lxml to parse this XML, and I am struggling to wrap my head around how to solve case 2. Case 1 and 3 work but when an object has all three cases then the output gets messed up.
I would like to receive following output:
Derived from sampling site: Peripheral blood
Transformant: NCBI-Taxonomy, 10376, Epstein-Barr virus (EBV)
Sequence variation: Hemizygous for FMR1 >200 CGG repeats (PubMed=25776194)
Monoclonal antibody target: UniProtKB, Q5T5X7, Human BEND3
Here is my very messy code which outpus the elements in wrong order. It worked fine for case 1 and 3, but when case 2 comes into play then the output is ordered wrong:
comment_cat = att.xpath('.//comment-list/comment/#category')
comment_text = att.xpath('.//comment-list/comment/text()')
cv_term = att.xpath('.//comment-list/comment/cv-term/text()')
xref = [a + ', ' + b for a,b in zip(att.xpath('.//comment-list/comment/xref-
list/xref/#database'),att.xpath('.//comment-list/comment/xref-list/xref/#accession'))]
property_list = att.xpath('.//comment-list/comment/xref-list/xref/property-list/property/#value')
xref_property_list = [a + ', ' + b for a,b in zip(xref, property_list)]
empty_str_in_text = ['\n ', '\n ', '\n ', '\n ']
comment_texts_all = cv_term+comment_text+xref_property_list
for e in empty_str_in_text:
if e in comment_texts_all:
comment_texts_all.remove(e)
key_values['Comments'] = ';; '.join([i + ': ' + j for i, j in zip(comment_cat,
comment_texts_all)])
Output:
Derived from sampling site: Epstein-Barr virus (EBV);;
Transformant: Peripheral blood ;;
Sequence variation: Hemizygous for FMR1 >200 CGG repeats (PubMed=25776194) ;;
Monoclonal antibody target: UniProtKB, Q5T5X7, Human BEND3

Here is a slightly alternative approach:
xml = '''<comment-list>
<comment category="Derived from sampling site"> Peripheral blood </comment>
<comment category="Transformant">
<cv-term terminology="NCBI-Taxonomy" accession="10376">Epstein-Barr virus (EBV)</cv-term>
</comment>
<comment category="Sequence variation"> Hemizygous for FMR1 >200 CGG repeats (PubMed=25776194)</comment>
<comment category="Monoclonal antibody target">
<xref-list>
<xref database="UniProtKB" category="Sequence databases" accession="Q5T5X7">
<property-list>
<property name="gene/protein designation" value="Human BEND3"/>
</property-list>
<url><![CDATA[https://www.uniprot.org/uniprot/Q5T5X7]]></url>
</xref>
</xref-list>
</comment>
<comment category="Knockout cell">
<method>KO mouse</method>
<xref-list>
<xref database="MGI" category="Organism-specific " accession="MGI:97740">
<property-list>
<property name="gene/protein designation" value="Polb"/>
</property-list>
<url><![CDATA[http://www.informatics.jax.org//MGI:97740]]></url>
</xref>
</xref-list>
</comment>
</comment-list>'''
from lxml import etree as ET
tree = ET.fromstring(xml)
result = ''
for comment in tree.iter('comment'):
result += f"{comment.get('category')}: "
cv_term = comment.find('cv-term')
xref_list = comment.find('xref-list')
method = comment.find('method')
if len(list(comment)) == 0:
result += comment.text
elif cv_term is not None:
result += ', '.join([cv_term.get('terminology'), cv_term.get('accession'), cv_term.text])
elif xref_list is not None and method is None:
result += ', '.join([xref_list.xpath('./xref/#database')[0], xref_list.xpath('./xref/#accession')[0], xref_list.xpath('./xref/property-list/property/#value')[0]])
elif method is not None:
result += method.text
result += '\n'
print(result)
Output:
Derived from sampling site: Peripheral blood
Transformant: NCBI-Taxonomy, 10376, Epstein-Barr virus (EBV)
Sequence variation: Hemizygous for FMR1 >200 CGG repeats (PubMed=25776194)
Monoclonal antibody target: UniProtKB, Q5T5X7, Human BEND3
Knockout cell: KO mouse

Remove duplicated sequences according to their ID names (bash or python)

I'm looking for a python or bash script in order to remove duplicated sequence by their ID.
Here is the input file:
>AJZ73152.1 hypothetical protein [Venturia canescens]
MAGLSTDKTDKTTVLLQYEVSHENYLARCIPGTRLHAKIHGSLPVLASSILTHNLDVKRADVFYLSGSSD
GSYYCDLPVSPQASKRVGDQTRETLRAQC
>AJZ73158.1 hypothetical protein [Venturia canescens]
MAGLSTDKTDKTTVLLQYEVSHENYLARCIPGTRLHAKIHGSLPVLASSILTHNLDVKRADVFYLSGSSD
GSYYCDLPVSPQASKRVGDQTRETLRAQC
>AKH40348.1 putative gp75-like protein [Kallithea virus]
MDQTDLLYTPQFEDYILEFCRAVSTDTTITAISPIIEVLKQSEYLRYLMKDPSNDSAKTCVRNFIVSKSH
LPQDFLYKFLAIVTMKISLAPSNVGFIHQSYNAKVIANNLQPTSRITNLTIAARQDQLRAESKNAITYVK
QTRMPPQILRMKFNDDLLPRCINAIGDLNQVIIEGNRSNGRDVGDFVRTVLK
>AKH40367.1 putative gp75-like protein [Kallithea virus]
MDQTDLLYTPQFEDYILEFCRAVSTDTTITAISPIIEVLKQSEYLRYLMKDPSNDSAKTCVRNFIVSKSH
LPQDFLYKFLAIVTMKISLAPSNVGFIHQSYNAKVIANNLQPTSRITNLTIAARQDQLRAESKNAITYVK
QTRMPPQILRMKFNDDLLPRCINAIGDLNQVIIEGNRSNGRDVGDFVRTVLK
>AZH40350.1 putative gp75-like protein [Kallithea virus]
MDQTDLLYTPQFEDYILEFCRAVSTDTTITAISPIIEVLKQSEYLRYLMKDPSNDSAKTCVRNFIVSKSH
LPQDFLYKFLAIVTMKISLAPSNVGFIHQSYNAKVIANNLQPTSRITNLTIAARQDQLRAESKNAITYVK
QTRMPPQILRMKFNDDLLPRCINAIGDLNQVIIEGNRSNGRDVGDFVRTVLK
>AKH40359.1 putative lef-4 [Kallithea virus]
MEEDNDNIPSTSLKVLNLLNIQVDTQQQQTVISNVANEHVANEYVSSEHVAIANEHVDNVNQSTTNAEFV
QKMPQTEVSMPTPTNPIYDEWESTIAIPITEEQYNIYKQKSHKSDVIFLFKNGTRLSCRTMQKKTTTYCR
NLISFYRNHWYPIRRTTAVESIEQLPPLYACDKVIFRLVVYHQNNIRISYNMEECAQGVKYNVEYEIEYK
RGISYREILIYERRLIRTVLQDNYEIKRQILSLVDLFSYVMTKVQMWHCFDPNKDYIWAYKWNGIKAKFL
ITDKLSDNGSNLTYIWPDANNITIEECHGNNISALVNFCFLVEIMDDCIVLIEAIGASIDQDIYTTEPAT
NSYVLKYLKDQNTSLKVGNKPVIIQEYYPPPLPNSYNREKFDGMIIVQDDMIIKWKIPTIDVKCIAPFKY
KIADDVLDFDFEGIPGKIYEISYKNEILRQRNDRIVASSPQEYAIFLESAKHLQ
>AKH40361.1 putative gp93-like protein [Kallithea virus]
MDLTLEHVTSWSCHLHSKETCVMKYYNGSYYHVIPVKNISVLAQTYNSQKIPDEFWEDLGPTPYMTAIYY
SDCVANVDMFRIILELFRNLDDSFLKFSTSNTPSDFIKRHIITDGIKRITLCNKHLLKSCKTKSNRPQTF
YTKDQWIKAILKGLFPKIDSSDKSGIPTNTPDWAIKLYPRGATSISAVTTPSSQMTHLAN
>AKH40366.1 putative gp93-like protein [Kallithea virus]
MDLTLEHVTSWSCHLHSKETCVMKYYNGSYYHVIPVKNISVLAQTYNSQKIPDEFWEDLGPTPYMTAIYY
SDCVANVDMFRIILELFRNLDDSFLKFSTSNTPSDFIKRHIITDGIKRITLCNKHLLKSCKTKSNRPQTF
YTKDQWIKAILKGLFPKIDSSDKSGIPTNTPDWAIKLYPRGATSISAVTTPSSQMTHLAN
>AKH40367.1 putative gp19-like protein [Kallithea virus]
MGVIRMTWNILSILITVIFVIALIWFVLYPTPIKYVLQCFVPKTEYEPNANYTTVKNYILYTNAKSNHTK
LIVIIPGGAGLLNSIANIYGFMNKLNETLGDDYDILTFSYPVRFKHTIRDSMLRVNEVLSDFTHYEEIHG
IGLSFGSLLLGAFNNKESNILSSQQMQVPQIGIKFKTFTGICGMYQPFFNVKLLTWLFDFYIMRGTPGIK
LYSCYGMPIPKLIITSNSDFLVSQSTKFLQSENAESLSYPTANLPHTFPQYINLPEAQQSIVKIVDFIKQ
NSN
>AKH40369.1 putative gp83-like protein [Kallithea virus]
MSESKLQHLHPEIINYYKSIKANGLKSPKMENNEEFITTLDRVEDDFKIPFISTYVLINNAYRHELSSNR
AKSIKQNIHAIREAKDVKIRTEVTAKVNKFEFIPSHFYTCSSKAIKVAVALFLRPAYTETLKRDFIFSLL
NHHSKTHTVSDVIDLCQKTIGDVRAFIKTVGNLNTTEKQRKQLICGLIECSELLRDRLCSKLAISVSLNG
YISLISLYLKHGHLKNVIPFEPLINLYVKESIAKCTQEEERVKILNQFKVDPVATIDDVIKGLPPAPNKV
SNSSTKSCVFKPDQNYQYYKGAPNYTRDIITTYHIEHGRRYRIQTYNDCLYDVLGYTLEAPNFLEATHSP
TTNGISAIEHEIYDRMSWSDRLNLIRFRTKIRIEDAKGSELNDYHGNSTDITISWFDDNEISCSKTISLK
KSDNKK
So the basic Idea is that I want to remove a sequence if its second name is duplicated.
For instance :
>AJZ73152.1 hypothetical protein [Venturia canescens]
MAGLSTDKTDKTTVLLQYEVSHENYLARCIPGTRLHAKIHGSLPVLASSILTHNLDVKRADVFYLSGSSD
GSYYCDLPVSPQASKRVGDQTRETLRAQC
First name= >AJZ73152.1
Second name= hypothetical protein [Venturia canescens]
So for the above example:
>AJZ73152.1 and >AJZ73158.1 have both the same second name "hypothetical protein [Venturia canescens]"
So I keep only one of them
>AKH40348.1, >AKH40367.1 and >AZH40350.1 have all the same second name "gp75-like protein [Kallithea virus]"
So I keep only one of them
>AKH40359.1, >AKH40367.1 and >AKH40369.1 have all no duplicate second name, I keel them all.
>AKH40366.1 and AKH40361.1 have both the same second name "putative gp93-like protein [Kallithea virus]"
So I keep only one of them
Here is what I should get as output:
>AJZ73152.1 hypothetical protein [Venturia canescens]
MAGLSTDKTDKTTVLLQYEVSHENYLARCIPGTRLHAKIHGSLPVLASSILTHNLDVKRADVFYLSGSSD
GSYYCDLPVSPQASKRVGDQTRETLRAQC
>AKH40348.1 putative gp75-like protein [Kallithea virus]
MDQTDLLYTPQFEDYILEFCRAVSTDTTITAISPIIEVLKQSEYLRYLMKDPSNDSAKTCVRNFIVSKSH
LPQDFLYKFLAIVTMKISLAPSNVGFIHQSYNAKVIANNLQPTSRITNLTIAARQDQLRAESKNAITYVK
QTRMPPQILRMKFNDDLLPRCINAIGDLNQVIIEGNRSNGRDVGDFVRTVLK
>AKH40359.1 putative lef-4 [Kallithea virus]
MEEDNDNIPSTSLKVLNLLNIQVDTQQQQTVISNVANEHVANEYVSSEHVAIANEHVDNVNQSTTNAEFV
QKMPQTEVSMPTPTNPIYDEWESTIAIPITEEQYNIYKQKSHKSDVIFLFKNGTRLSCRTMQKKTTTYCR
NLISFYRNHWYPIRRTTAVESIEQLPPLYACDKVIFRLVVYHQNNIRISYNMEECAQGVKYNVEYEIEYK
RGISYREILIYERRLIRTVLQDNYEIKRQILSLVDLFSYVMTKVQMWHCFDPNKDYIWAYKWNGIKAKFL
ITDKLSDNGSNLTYIWPDANNITIEECHGNNISALVNFCFLVEIMDDCIVLIEAIGASIDQDIYTTEPAT
NSYVLKYLKDQNTSLKVGNKPVIIQEYYPPPLPNSYNREKFDGMIIVQDDMIIKWKIPTIDVKCIAPFKY
KIADDVLDFDFEGIPGKIYEISYKNEILRQRNDRIVASSPQEYAIFLESAKHLQ
>AKH40361.1 putative gp93-like protein [Kallithea virus]
MDLTLEHVTSWSCHLHSKETCVMKYYNGSYYHVIPVKNISVLAQTYNSQKIPDEFWEDLGPTPYMTAIYY
SDCVANVDMFRIILELFRNLDDSFLKFSTSNTPSDFIKRHIITDGIKRITLCNKHLLKSCKTKSNRPQTF
YTKDQWIKAILKGLFPKIDSSDKSGIPTNTPDWAIKLYPRGATSISAVTTPSSQMTHLAN
>AKH40367.1 putative gp19-like protein [Kallithea virus]
MGVIRMTWNILSILITVIFVIALIWFVLYPTPIKYVLQCFVPKTEYEPNANYTTVKNYILYTNAKSNHTK
LIVIIPGGAGLLNSIANIYGFMNKLNETLGDDYDILTFSYPVRFKHTIRDSMLRVNEVLSDFTHYEEIHG
IGLSFGSLLLGAFNNKESNILSSQQMQVPQIGIKFKTFTGICGMYQPFFNVKLLTWLFDFYIMRGTPGIK
LYSCYGMPIPKLIITSNSDFLVSQSTKFLQSENAESLSYPTANLPHTFPQYINLPEAQQSIVKIVDFIKQ
NSN
>AKH40369.1 putative gp83-like protein [Kallithea virus]
MSESKLQHLHPEIINYYKSIKANGLKSPKMENNEEFITTLDRVEDDFKIPFISTYVLINNAYRHELSSNR
AKSIKQNIHAIREAKDVKIRTEVTAKVNKFEFIPSHFYTCSSKAIKVAVALFLRPAYTETLKRDFIFSLL
NHHSKTHTVSDVIDLCQKTIGDVRAFIKTVGNLNTTEKQRKQLICGLIECSELLRDRLCSKLAISVSLNG
YISLISLYLKHGHLKNVIPFEPLINLYVKESIAKCTQEEERVKILNQFKVDPVATIDDVIKGLPPAPNKV
SNSSTKSCVFKPDQNYQYYKGAPNYTRDIITTYHIEHGRRYRIQTYNDCLYDVLGYTLEAPNFLEATHSP
TTNGISAIEHEIYDRMSWSDRLNLIRFRTKIRIEDAKGSELNDYHGNSTDITISWFDDNEISCSKTISLK
KSDNKK

This code should do what you need. Please let me know if it works for you :
awk 'BEGIN{RS="";FS="\n";}
{
split($1,descs," ");
for(i=2; i <= length(descs); i++){
second_names[NR] = second_names[NR] " " descs[i]
}
already_seen = 0
for(c=1; c <= length(seen_names); c++ ) {
if (second_names[NR] == seen_names[c]) {
already_seen = 1
}
}
if(already_seen == 0) {
print $0
print "\n"
seen_names[length(seen_names) + 1] = second_names[NR];
}
}
' input.txt
Regards!

Simple method:
If you don't care about ordering you can simply place all of them in a dictionary where id (second name) is key. That way you are unable to have duplicates but output will not be ordered and in every duplicate only the last entry will be kept. This is as simple as parsing each unit into 3 parts and adding them into a variable.
More complex method:
Create an empty list and go over each unit one by one and check if you need to add it to list or not. This preserves ordered input and allows you to decide which duplicate you will keep.

extracting strings using regular expression

I have the following strings:
LOW QUALITY PROTEIN: cysteine proteinase 5-like [Solanum pennellii]
PREDICTED: LOW QUALITY PROTEIN: uncharacterized protein LOC107059219 [Solanum pennellii]
XP_019244624.1 PREDICTED: peroxidase 40-like [Nicotiana attenuata]
RVW92024.1 Retrovirus-related Pol polyprotein from transposon TNT 1-94 [Vitis vinifera]
hypothetical protein VITISV_035070 [Vitis vinifera]
How to extract the below strings from the above strings?
cysteine proteinase 5-like
uncharacterized protein LOC107059219
peroxidase 40-like
Retrovirus-related Pol polyprotein from transposon TNT 1-94
hypothetical protein VITISV_035070

I think this problem don't need regex. I would prefer following solution because it is easy to understand
st = "PREDICTED: LOW QUALITY PROTEIN: uncharacterized protein LOC107059219 [Solanum pennellii]"
st.split(":")[-1].split("[")[0].strip()

splitting strings after certain characters

I would like to split my string after certain characters are found.
identifier = filecontent_id[0].split("SV=")[0]
I have this, but this "deletes" everything before "SV=" and I would like for it to "delete" everything 1 character after it. For example, it would "delete" everything after "SV=1" but I did not put 1 there because it doesn't always equal 1. The string is:
>tr|A0A024RAP8|A0A024RAP8_HUMAN HCG2009644, isoform CRA_b OS=Homo sapiens GN=KLRC4-KLRK1 PE=4 SV=1MGWIRGRRSRHSWEMSEFHNYNLDLKKSDFSTRWQ
and I am trying to only get:
>tr|A0A024RAP8|A0A024RAP8_HUMAN HCG2009644, isoform CRA_b OS=Homo sapiens GN=KLRC4-KLRK1 PE=4 SV=1

A regex might be better, but the below works
SPLIT="SV="
line=">tr|A0A024RAP8|A0A024RAP8_HUMAN HCG2009644, isoform CRA_b OS=Homo sapiens GN=KLRC4-KLRK1 PE=4 SV=1MGWIRGRRSRHSWEMSEFHNYNLDLKKSDFSTRWQ"
print line.split(SPLIT)[0] + SPLIT + line.split(SPLIT)[1][0]

How to write multiple lines in a single line?

How can I write multiple lines in a single line? My inputs are like this:
HOXC11
HOXC11, HOX3H, MGC4906
human, Homo sapiens
HOXB6
HOXB6, HOX2, HU-2, HOX2B, Hox-2.2
human, Homo sapiens
HOXB13
HOXB13
human, Homo sapiens
PAX5
PAX5, BSAP
human, Homo sapiens
I need to make it into a single line like this:
HOXC11 HOXC11, HOX3H, MGC4906 human, Homo sapiens
HOXB6 HOXB6, HOX2, HU-2, HOX2B, Hox-2.2 human, Homo sapiens
HOXB13 HOXB13 human, Homo sapiens

Assuming your input is from a file, let's call it homosapiens.txt, you can go from the specified input to the desired output as follow:
with open('homosapiens.txt', 'r') as f:
for line in f:
if line == 'human, Homo sapiens':
print line # this will print and go to a newline
elif line:
print line, # the comma after line suppresses the newline

textInput = textInput.rstrip('\n')