I'm aware that you can use
df1 = df1[df1['Computer Name'] != 'someNameToBeDropped']
to drop a given string as a row
what if i wanted to do it the other way around. Let's say dropping everything except what i have in a list of strings.
is there a simple hack I haven't noticed?
Try this to get rows such that value of col is in that given list
df = df[df[column].isin(list_of_strings)]
Additional to exclude what's in the list
df = df[~df[column].isin(list_of_values)]
Related
I'm using the .at function to try and save all columns under one header in a list.
The file contains entries for country and population.
df = pandas.read_csv("file.csv")
population_list = []
df2 = df[df['country'] == "India"]
for i in range(len(df2)):
population_list = df2.at[i, 'population']
This is throwing a KeyError. However, the df.at seems to be working fine for the original dataframe. Is .at just not allowed in this case?
IIUC, you don't need to loop over your dataframe to get what you need. Simply use:
population_list = df2["population"].tolist()
If you really want to use the loop (not recommended when unnecessary), note that the index has likely changed after your filter, i.e not consecutive integers.
Try:
for i in df2.index:
population_list.append(df2.at[i, 'population'])
Note: In your code you keep trying to reassign the entire list to one value instead of appending.
In at you pass the index value and the column name.
In the case of the "original" DataFrame all is OK, because probably the index contains
consecutive values starting from 0.
But when you run df2 = df[df['country'] == "India"] then df2 contains
only a subset of original rows, so the index does not contain consecutive numbers.
One of possible solutions is to run reset_index() on df2.
Then the index will again contain consecutive numbers and your code should raise no exception.
Edit
But your code raises other doubts.
Remember that at returns a single value, taken from a cell
with particular index value and column, not a list.
So maybe it is enough to run:
population_India = df.set_index('country').at['India', 'population']
You don't need any list. You want to find just the popupation of India, a single value.
I have some data that changes regularly but the column headers need to be consistent (so I cant drop the headers) but I need to clear our the strings in a given column.
This is what I have now but this only seems to work for where I know what the string is called and one at a time?
df1= pd.read_csv(r'C:\Users\Test.csv')
df2 = df1.drop(df1[~(df1['column'] != 'String1')].index)
You can use the pd.drop function which removes rows having a specific index from a dataframe.
for i in df.index:
if type(df.loc[i, 'Aborted Reason']) == str:
df.drop(i, inplace = True)
df.drop will remove the index having a string in the relevant column from the dataframe.
I am trying to insert or add from one dataframe to another dataframe. I am going through the original dataframe looking for certain words in one column. When I find one of these terms I want to add that row to a new dataframe.
I get the row by using.
entry = df.loc[df['A'] == item]
But when trying to add this row to another dataframe using .add, .insert, .update or other methods i just get an empty dataframe.
I have also tried adding the column to a dictionary and turning that into a dataframe but it writes data for the entire row rather than just the column value. So is there a way to add one specific row to a new dataframe from my existing variable ?
So the entry is a dataframe containing the rows you want to add?
you can simply concatenate two dataframe using concat function if both have the same columns' name
import pandas as pd
entry = df.loc[df['A'] == item]
concat_df = pd.concat([new_df,entry])
pandas.concat reference:
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html
The append function expect a list of rows in this formation:
[row_1, row_2, ..., row_N]
While each row is a list, representing the value for each columns
So, assuming your trying to add one row, you shuld use:
entry = df.loc[df['A'] == item]
df2=df2.append( [entry] )
Notice that unlike python's list, the DataFrame.append function returning a new object and not changing the object called it.
See also enter link description here
Not sure how large your operations will be, but from an efficiency standpoint, you're better off adding all of the found rows to a list, and then concatenating them together at once using pandas.concat, and then using concat again to combine the found entries dataframe with the "insert into" dataframe. This will be much faster than using concat each time. If you're searching from a list of items search_keys, then something like:
entries = []
for i in search_keys:
entry = df.loc[df['A'] == item]
entries.append(entry)
found_df = pd.concat(entries)
result_df = pd.concat([old_df, found_df])
I am working with a pandas dataframe where i want to group by one column, grab the last row of each group (creating a new dataframe), and then drop those rows from the original.
I've done a lot of reading and testing, and it seems that I can't do that as easily as I'd hoped. I can do a kludgy solution, but it seems inefficient and, well, kludgy.
Here's pseudocode for what I wanted to do:
df = pd.DataFrame
last_lines = df.groupby('id').last()
df.drop(last_lines.index)
creating the last_lines dataframe is fine, it's dropping those rows from the original df that's an issue. the problem is that the original index (from df) is disconnected when last_lines is created. i looked at filter and transform, but neither seems to address this problem. is there a good way to split the dataframe into two pieces based on position?
my kludge solution is to iterate over the group iterator and create a list of indexes, then drop those.
grouped = df.groupby('id')
idx_to_remove = []
for _, group in grouped:
idx_to_remove.append(group.tail(1).index[0])
df.drop(idx_to_remove)
Better suggestions?
If you use .reset_index() first, you'll get the index as a column and you can use .last() on that to get the indices you want.
last_lines = df.reset_index().groupby('A').index.last()
df.drop(last_lines)
Here the index is accessed as .index because "index" is the default name given to this column when you use reset_index. If your index has a name, you'll use that instead.
You can also "manually" grab the last index by using .apply():
last_lines = d.groupby('A').apply(lambda g: g.index[-1])
You'll probably have to do it this way if you're using a MultiIndex (since in that case using .reset_index() would add multiple columns that can't easily be combined back into indices to drop).
Try:
df.groupby('A').apply(lambda x: x.iloc[:-1, :])
I'm trying to remove the percent sign after a value in a pandas dataframe, relevant code:
for i in loansdata:
if i.endswith('%'):
i = i[:-1]
I was thinking that i = i[:-1] would set the new value, but it doesn't. How do I go about it? For clarity: if I print i inside the for loop, it prints without the percent sign. But if I print the whole dataframe, it has not changed.
use str.replace to replace a specific character for a column:
df[col] = df[col].str.replace('%','')
What you're doing depending on what loansdata actually is, is either looping over the columns or the row values of a column.
You can't modify the row contents like that, even if you could you should avoid loops where a vectorised solution exists.
If % exists in multiple cols then you could call the above for each col but this method only exists for str dtypes