Develop Reference

How to select columns from different tables based on other facture to create a new dataframe python

I have 2 DataFrames both countain countries
1-first have 183 row
2-the second have 156 row
both of them has import information on each other
I need one column from the first and one column from the second
My goal is to create a single Dataframe contain both columns that I need and name of the contain that both datafames commo.
This is what I did and the message that I got
for i in range(183) :
for j in range(156):
if df['Country'][i]==df_happy['Country or region'][j]:
df.drop(i,axis=0,inplace=True)
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-25-e078ef71e219> in <module>
1 for i in range(183) :
2 for j in range(156):
----> 3 if df['Country'][i]==df_happy['Country or region'][j]:
4 df.drop(i,axis=0,inplace=True)
/opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages/pandas/core/series.py in __getitem__(self, key)
869 key = com.apply_if_callable(key, self)
870 try:
--> 871 result = self.index.get_value(self, key)
872
873 if not is_scalar(result):
/opt/conda/envs/Python-3.7-main/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_value(self, series, key)
4403 k = self._convert_scalar_indexer(k, kind="getitem")
4404 try:
-> 4405 return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
4406 except KeyError as e1:
4407 if len(self) > 0 and (self.holds_integer() or self.is_boolean()):
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()
KeyError: 1

You can merge both data frames:
newdf=df.merge(df_happy,how='left', left_on='Country', right_on='Country or region')
and then drop the extra columns with:
newdf.drop(columns=['B', 'C'])

Why does DataFrame row selection syntax df[:2] work but not if the syntax df[1]?

I have following data:
data = pd.DataFrame(np.arange(16).reshape(4, 4), index = ['Ohio', 'Colorado', 'Utah', 'New York'], columns = ['one', 'two', 'three', 'four'])
If I run:
data[:2]
the output will be:
one two three four
Ohio 0 1 2 3
Colorado 4 5 6 7
If I run: data[1], the following error will show up:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
2645 try:
-> 2646 return self._engine.get_loc(key)
2647 except KeyError:
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
KeyError: 1
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
<ipython-input-81-c402bf503b75> in <module>
----> 1 data[1]
~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py in __getitem__(self, key)
2798 if self.columns.nlevels > 1:
2799 return self._getitem_multilevel(key)
-> 2800 indexer = self.columns.get_loc(key)
2801 if is_integer(indexer):
2802 indexer = [indexer]
~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
2646 return self._engine.get_loc(key)
2647 except KeyError:
-> 2648 return self._engine.get_loc(self._maybe_cast_indexer(key))
2649 indexer = self.get_indexer([key], method=method, tolerance=tolerance)
2650 if indexer.ndim > 1 or indexer.size > 1:
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
KeyError: 1
Why could I run data[:2] but not data[1]? It doesn't make sense to me. Thank you in advance:-)

[ ] - indexing operator looks for a column name entered.
In your example data[1], there is no column by name 1. So the key error.
But, when you pass slicing notation : inside the indexing operator, the indexing operator changes the behavior from "searching for columns" to "searching for rows based on the range"

The first portion data[:1] is a slicing operation. and As you set the index = ['Ohio', 'Colorado', 'Utah', 'New York'] so there is no default index (0-9..) that's why it gives you a key error.
if you enter the column name. like data['one'] you will get
Ohio 0
Colorado 4
Utah 8
New York 12
Name: one, dtype: int64

Pandas - selecting several rows and columns with date as key

I try to select several specific rows and columns from this dataframe:
Open High Low Close Volume Dividends Stock Splits
Date
2020-07-17 387.95 388.59 383.36 385.31 23046700 0 0
2020-07-20 385.67 394.00 384.25 393.43 22579500 0 0
2020-07-21 396.69 397.00 386.97 388.00 25911500 0 0
2020-07-22 386.77 391.90 386.41 389.09 22215400 0 0
2020-07-23 387.99 388.31 384.25 385.17 4554225 0 0
It is possible to select some rows which are following each other with one specific column
hist["2020-07-20":"2020-07-22"]["Close"]
Date
2020-07-20 393.43
2020-07-21 388.00
2020-07-22 389.09
Name: Close, dtype: float64
When i try more columns which are following - i get this error:
hist["2020-07-20":"2020-07-22", "Open":"Close"]
TypeError Traceback (most recent call last)
<ipython-input-25-57b43e76004f> in <module>
----> 1 hist["2020-07-20":"2020-07-22", "Open":"Close"]
c:\users\polzi\appdata\local\programs\python\python37\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
2798 if self.columns.nlevels > 1:
2799 return self._getitem_multilevel(key)
-> 2800 indexer = self.columns.get_loc(key)
2801 if is_integer(indexer):
2802 indexer = [indexer]
c:\users\polzi\appdata\local\programs\python\python37\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
2644 )
2645 try:
-> 2646 return self._engine.get_loc(key)
2647 except KeyError:
2648 return self._engine.get_loc(self._maybe_cast_indexer(key))
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
TypeError: '(slice('2020-07-20', '2020-07-22', None), slice('Open', 'Close', None))' is an invalid key
I also tried to select several rows which are NOT following - does not work either
hist["2020-07-20","2020-07-22"]["Low"]
KeyError Traceback (most recent call last)
c:\users\polzi\appdata\local\programs\python\python37\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
2645 try:
-> 2646 return self._engine.get_loc(key)
2647 except KeyError:
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
KeyError: ('2020-07-20', '2020-07-22')
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
<ipython-input-26-aefccd2025a5> in <module>
----> 1 hist["2020-07-20","2020-07-22"]["Low"]
c:\users\polzi\appdata\local\programs\python\python37\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
2798 if self.columns.nlevels > 1:
2799 return self._getitem_multilevel(key)
-> 2800 indexer = self.columns.get_loc(key)
2801 if is_integer(indexer):
2802 indexer = [indexer]
c:\users\polzi\appdata\local\programs\python\python37\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
2646 return self._engine.get_loc(key)
2647 except KeyError:
-> 2648 return self._engine.get_loc(self._maybe_cast_indexer(key))
2649 indexer = self.get_indexer([key], method=method, tolerance=tolerance)
2650 if indexer.ndim > 1 or indexer.size > 1:
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
KeyError: ('2020-07-20', '2020-07-22')
How can i select several specific rows and columns - which are not following after each other?

hist[["Open","High","Low","Close"]]["2020-07-20":"2020-07-22"]
will give you a dataframe with the preselected columns.
you can also use :
hist[hist.columns[0:4]]["2020-07-20":"2020-07-22"]
If the rows are not following, you can use :
hist[hist.index.isin(["2020-07-20","2020-07-22"])][hist.columns[0:4]]
if both rows and columns are arbitrary, you can use
hist[hist.index.isin(["2020-07-20","2020-07-22"])][["Open","Close"]]

Pandas dataframe index causing problems when indexing subset of dataframe. How do I remove the indexes, or prevent the error from occurring?

I have a dataframe x1. I made a subset of the dataframe, x1_sub, where I need to use a for loop to index its items. But because the subset retains the indexing of the original pandas dataframe, it has its rows like so:
x1_sub['words']
1 investment
2 fund
4 company
7 claim
9 customer
20 easy
... ...
So, when I do something like this to index the rows of x1_sub serially:
for i in range(len(x1)):
for j in range(len(x1_sub)):
if (x1['word'][i]==x1_sub['word'][j]):
print(i, j)
it gives the following error:
KeyError Traceback (most recent call last)
<ipython-input-48-e3c9806732a6> in <module>()
3 for i in range(len(x1)):
4 for j in range(len(x1_sub)):
----> 5 if (x1['word'][i]==x1_sub['word'][j]):
6 print(i, j)
7
c:\users\h473\appdata\local\programs\python\python35\lib\site-packages\pandas\core\series.py in __getitem__(self, key)
621 key = com._apply_if_callable(key, self)
622 try:
--> 623 result = self.index.get_value(self, key)
624
625 if not is_scalar(result):
c:\users\h473\appdata\local\programs\python\python35\lib\site-packages\pandas\core\indexes\base.py in get_value(self, series, key)
2558 try:
2559 return self._engine.get_value(s, k,
-> 2560 tz=getattr(series.dtype, 'tz', None))
2561 except KeyError as e1:
2562 if len(self) > 0 and self.inferred_type in ['integer', 'boolean']:
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_value()
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_value()
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()
KeyError: 0
EDIT: Some example data:
The following data is saved in a csv file named example.csv:
word score
service 1
customer 4
agent 3
product 6
easy 2
claim 2
fast 1
financial 5
information 1
benefit 4
company 3
helpful 6
time 2
future 2
policy 1
health 5
life 1
fund 4
complicated 3
investment 6
join 2
payment 2
premium 1
excellent 5
experience 1
family 4
nice 3
proces 6
satisfactory 2
And the code is this:
import pandas as pd
x1 = pd.read_csv(r'C:\Users\h473\Documents\Indonesia_verbatims W1 2018\Insurance Data X3\example.csv')
x1_sub = x1[x1['score']<=2]
for i in range(len(x1)):
for j in range(len(x1_sub)):
if (x1['word'][i]==x1_sub['word'][j]):
print(i, j)
And this is the output:
0 0
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-63-08d55a712c99> in <module>()
7 for i in range(len(x1)):
8 for j in range(len(x1_sub)):
----> 9 if (x1['word'][i]==x1_sub['word'][j]):
10 print(i, j)
c:\users\h473\appdata\local\programs\python\python35\lib\site-packages\pandas\core\series.py in __getitem__(self, key)
621 key = com._apply_if_callable(key, self)
622 try:
--> 623 result = self.index.get_value(self, key)
624
625 if not is_scalar(result):
c:\users\h473\appdata\local\programs\python\python35\lib\site-packages\pandas\core\indexes\base.py in get_value(self, series, key)
2558 try:
2559 return self._engine.get_value(s, k,
-> 2560 tz=getattr(series.dtype, 'tz', None))
2561 except KeyError as e1:
2562 if len(self) > 0 and self.inferred_type in ['integer', 'boolean']:
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_value()
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_value()
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()
KeyError: 1
EDIT 2: Also, if x1_sub is a list, then the error is different:
import pandas as pd
x1 = pd.read_csv(r'C:\Users\h473\Documents\Indonesia_verbatims W1 2018\Insurance Data X3\example.csv')
#x1_sub = x1[x1['score']<=2]
x1_sub = ['service', 'claim', 'health', 'fund', 'premium', 'nice', 'process']
for i in range(len(x1)):
for j in range(len(x1_sub)):
if (x1['word'][i]==x1_sub['word'][j]):
print(i, j)
Produces the following output:
TypeError Traceback (most recent call last)
<ipython-input-68-dec8c7e33757> in <module>()
8 for i in range(len(x1)):
9 for j in range(len(x1_sub)):
---> 10 if (x1['word'][i]==x1_sub['word'][j]):
11 print(i, j)
TypeError: list indices must be integers or slices, not str

I think looping is best avoid in pandas, because very slow if exist some vectorized solution:
x1_sub = ['service', 'claim', 'health', 'fund', 'premium', 'nice', 'process']
x2 = x1[x1['word'].isin(x1_sub)]
print (x2)
word score
0 service 1
5 claim 2
15 health 5
17 fund 4
22 premium 1
26 nice 3

try to assign separate indices by using Dataframe.set_index(keys,inplace=True)
please refer to this documentation
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.set_index.html

Numpy TypeError: an integer is required

This will be maybe quite personal question but I don't know who to ask I hope somebody can help and don't skip me THANKS!. I have installed python using Anaconda and using Jupyter notebook. I have 2 csv files of data.
products.head()
ID_FUPID FUPID
0 1 674563
1 2 674597
2 3 674606
3 4 694776
4 5 694788
Products contain id of product and product number.
ratings.head()
ID_CUSTOMER ID_FUPID RATING
0 1 216 1
1 2 390 1
2 3 851 5
3 4 5897 1
4 5 9341 1
Ratings containt id of customer, productID and Rating which customer give to product.
I have created table as:
M = ratings.pivot_table(index=['ID_CUSTOMER'],columns=['ID_FUPID'],values='RATING')
Which is showing data correctly in matrix with productID= columns and customerID as rows.
I wanted to count pearson colleration between products so here is the pearson function:
def pearson(s1, s2):
import numpy as np
"""take two pd.series objects and return a pearson correlation"""
s1_c = s1 - s1.mean()
s2_c = s2 - s2.mean()
return np.sum(s1_c * s2_c) / np.sqrt(np.sum(s1_c ** 2) * np.sum(s2_c ** 2))
When I'm trying to count pearson(M['17'], M['21']) I got following errors:
TypeError Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()
TypeError: an integer is required
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
2441 try:
-> 2442 return self._engine.get_loc(key)
2443 except KeyError:
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
KeyError: '17'
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call last)
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()
TypeError: an integer is required
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
<ipython-input-277-d4ead225b6ab> in <module>()
----> 1 pearson(M['17'], M['21'])
~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
1962 return self._getitem_multilevel(key)
1963 else:
-> 1964 return self._getitem_column(key)
1965
1966 def _getitem_column(self, key):
~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py in _getitem_column(self, key)
1969 # get column
1970 if self.columns.is_unique:
-> 1971 return self._get_item_cache(key)
1972
1973 # duplicate columns & possible reduce dimensionality
~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\generic.py in _get_item_cache(self, item)
1643 res = cache.get(item)
1644 if res is None:
-> 1645 values = self._data.get(item)
1646 res = self._box_item_values(item, values)
1647 cache[item] = res
~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals.py in get(self, item, fastpath)
3588
3589 if not isnull(item):
-> 3590 loc = self.items.get_loc(item)
3591 else:
3592 indexer = np.arange(len(self.items))[isnull(self.items)]
~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\indexes\base.py in get_loc(self, key, method, tolerance)
2442 return self._engine.get_loc(key)
2443 except KeyError:
-> 2444 return self._engine.get_loc(self._maybe_cast_indexer(key))
2445
2446 indexer = self.get_indexer([key], method=method, tolerance=tolerance)
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas\_libs\index.pyx in pandas._libs.index.IndexEngine.get_loc()
KeyError: '17'
I will really appreciate any help ! thanks a million.

There were two places in the error message with the following line:
KeyError: '17'
This indicates there is no key '17' in M. This is likely because your index is an integer. However, you are currently accessing the DataFrame M with a string. The code to call pearson might be as follows:
pearson(M[17], M[21])