Problem with removing redundancy from a file [duplicate] - python

This question already has answers here:
drop_duplicates not working in pandas?
(7 answers)
DataFrame.drop_duplicates and DataFrame.drop not removing rows
(2 answers)
Closed 3 years ago.
I've got a DataSet with two columns, one with categorical value (State2), and another (State) that contains the same values only in binary.
I used OneHotEncoding.
import pandas as pd
mydataset = pd.read_csv('fieldprotobackup.binetflow')
mydataset.drop_duplicates(['Proto2','Proto'], keep='first')
mydataset.to_csv('fieldprotobackup.binetflow', columns=['Proto2','Proto'], index=False)
Dataset
I'd like to remove all redundancies from the file. While researching, I found the command df.drop_duplicates, but it's not working for me.


You either need to add the inplace=True parameter, or you need to capture the returned dataframe:
mydataset.drop_duplicates(['Proto2','Proto'], keep='first', inplace=True)
or
no_duplicates = mydataset.drop_duplicates(['Proto2','Proto'], keep='first')
Always a good idea to check the documentation when something isn't working as expected.

Related

Pandas column separation using .loc [duplicate]

This question already has answers here:
Selecting multiple columns in a Pandas dataframe
(22 answers)
Closed last month.
I am an amateur user.
I watched many videos but I couldn't figure out this error.
How can I keep PERSON_WGHT, LOS, and IDC_DC_CD_1 as a columns for all rows that is 386816.

If you need to select multip0le columns from all the records then use df[[column_list]].
df_new = df[['PERSON_WGHT', 'LOS', 'IDC_DC_CD_1']]

how to fix groupby elimination of index? [duplicate]

This question already has answers here:
Pandas reset index is not taking effect [duplicate]
(4 answers)
Closed 3 months ago.
Please see images -
After creating a dataframe, I use groupby, then I reset the index column only to find that the column for 'county' is still unseen by the dataframe. Please help to rectify.

The df.reset_index() by default is not an "inplace" operation. But with use of the inplace parameter you can make it behave as such.
1. Either use inplace=True -
mydf.reset_index(inplace=True)
2. Or save the df into another (or the same) variable -
mydf = mydf.reset_index()
This should fix your issue.

In pandas, how to turn a dataframe into tidy data? [duplicate]

This question already has answers here:
How do I melt a pandas dataframe?
(3 answers)
Convert columns into rows with Pandas
(6 answers)
Closed last year.
In python and pandas I have a dataframe that I need to turn into tidy data to make charts easier
The original data is like this:
I want to transform into a dataframe, with the transposition of the data and adapting column names:
Please is there a way in python to do this?

Use melt:
out = df.melt('year', var_name='localization', value_name='number_of_tests')
You can also use:
out = df.set_index('year').rename_axis(columns='localization').unstack() \
.rename('number_of_tests').reset_index()

Final value in dataframe showing as not inside dataframe when going through for loop [duplicate]

This question already has answers here:
Python Pandas -- why does the `in` operator work with indices and not with the data?
(1 answer)
How to determine whether a Pandas Column contains a particular value
(11 answers)
Closed 1 year ago.
I'm checking to see if all the values in one column of one dataframe lies in the column of another dataframe. When I run the code below, it says that 4 does not exist in df1. Is there any particular reason for this?
list1=[1,2,3,4]
list2=[1,2,3,4]
df2=pd.DataFrame(list2)
df2.rename(columns={0:"List2"},inplace=True)
df1=pd.DataFrame(list1)
df1.rename(columns={0:"List1"},inplace=True)
for i in df2['List2']:
if i not in df1['List1']:
print(i)

how can I remove index(auto made) in pandas.read_csv [duplicate]

This question already has answers here:
Removing index column in pandas when reading a csv
(9 answers)
Closed 3 years ago.
here is my dataframe
my csv file is
date,open,high,low,close,volume,cap,Unnamed: 7
20190816,28600,28850,28150,28350,335508,6065213000000,
20190814,29550,29600,28800,28950,296026,6193563000000,
20190813,29400,29900,29400,29550,196955,6321927000000,
20190812,29450,30350,29400,29850,166580,6386109000000,
20190809,29500,30300,29450,29750,468338,6364715000000,
20190808,29000,30000,29000,29650,448959,6343321000000,
20190807,29800,29800,28950,29000,431524,6204260000000,
20190806,30900,30950,29650,29900,710348,6396806000000,
20190805,30300,31100,30300,30950,608970,6621443000000,
20190802,30400,30750,29900,30400,420984,6503776000000,
I don't know why 0 ~ 11 index exists
I want to remove this (0~11)
I searched and tried index_col=False, index_col=None and to_csv with index=False but The problem was not resolved.
how can I remove this index(0~11)?
Your valuable opinions and thoughts will be very much appreciated.

The only solution that fully matches what you desire is to create a string:
print(df.to_string(index=False))
Another solution would be the below, it will still be a dataframe, but just the first column's values will be shifted down one level:
print(df.set_index('date'))

You cannot remove the index of a Pandas DataFrame. It is not one of the columns of your DataFrame. And it is not coming from the csv file.
See: https://stackoverflow.com/a/20107825/4936825

You can not display this automatically generated index with „df.style.hide_index()“ or set one of the colums as index with set_index() method.

Categories

Resources