Develop Reference

How to validate a dataframe index using SchemaModel in Pandera

I can validate a DataFrame index using the DataFrameSchema like this:
import pandera as pa
from pandera import Column, DataFrameSchema, Check, Index
schema = DataFrameSchema(
columns={
"column1": pa.Column(int),
},
index=pa.Index(int, name="index_name"),
)
# raises the error as expected
schema.validate(
pd.DataFrame({"column1": [1, 2, 3]}, index=pd.Index([1, 2, 3], name="index_incorrect_name"))
)
Is there a way to do the same using a SchemaModel?

You can do as follows -
import pandera as pa
from pandera.typing import Index, Series
class Schema(pa.SchemaModel):
idx: Index[int] = pa.Field(ge=0, check_name=True)
column1: Series[int]
df = pd.DataFrame({"column1": [1, 2, 3]}, index=pd.Index([1, 2, 3], name="index_incorrect_name"))
Schema.validate(df)

Found an answer in GitHub
You can use pa.typing.Index to type-annotate an index.
class Schema(pa.SchemaModel):
column1: pa.typing.Series[int]
index_name: pa.typing.Index[int] = pa.Field(check_name=True)
See how you can validate a MultiIndex index: https://pandera.readthedocs.io/en/stable/schema_models.html#multiindex

Plotting names with Chinese letters throws a error

I have a problem. I want to print the 5 most names. But unfortunately the names are not only Latin letters, but also Chinese letters. As soon as I want to print the plot, I got:
C:\Users\user\Anaconda3\lib\site-packages\matplotlib\backends\backend_agg.py:240: RuntimeWarning: Glyph 32422 missing from current font.
How can I solve this error?
import pandas as pd
import seaborn as sns
d = {'id': [1, 2, 3, 4, 5],
'name': ['Max Power', 'Jessica', '约翰·多伊', '哈拉尔量杯', 'Frank High'],
}
df = pd.DataFrame(data=d)
print(df)
df_count = df['name'].value_counts()[:5]
ax = sns.barplot(x=df_count.index, y=df_count)

module 'tsfresh.feature_extraction.feature_calculators' has no attribute

I'm new using tsfresh, when I use the following lines, I get the extracted feature as desired
import numpy as np
import pandas as pd
from tsfresh.feature_extraction import ComprehensiveFCParameters
from tsfresh import extract_features
df = pd.DataFrame(np.array([[1, 2, 3, 4],[1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]]),
columns=['Context ID','Time Elapsed', 'time_serie A', 'time_serie B'])
settings = ComprehensiveFCParameters()
kind_to_fc_parameters = {
"time_serie A": {},
"time_serie B": {"mean": None}
}
extract_features = extract_features(df, kind_to_fc_parameters =kind_to_fc_parameters,
column_id='Context ID', column_sort="Time Elapsed")
extract_features
However, when I change {"mean": None} by {"absolute_maximum": None} or "count_above": [{"t": 0.05}] it'won't work anymore:
module 'tsfresh.feature_extraction.feature_calculators' has no
attribute 'absolute_maximum'
What do I miss ?

I just had a similar issue with another calculation I chose and found it's just not in the feature_calculators.py (you can open it from yourdirectory\Python\Python37\Lib\site-packages\tsfresh\feature_extraction), so I did pip install tsfresh -U in terminal to get the latest tsfresh, checked feature_calculators.py again, my desired function is there and code runs fine then.

Python ValueError from np.where create flag based on one condition

If the city has been mentioned in cities_specific I would like to create a flag in the cities_all data. It's just a minimal example and in reality I would like to create multiple of these flags based on multiple data frames. That's why I tried to solve it with isin instead of a join.
However, I am running into ValueError: Length of values (3) does not match length of index (7).
# import packages
import pandas as pd
import numpy as np
# create minimal data
cities_specific = pd.DataFrame({'city': ['Melbourne', 'Cairns', 'Sydney'],
'n': [10, 4, 8]})
cities_all = pd.DataFrame({'city': ['Vancouver', 'Melbourne', 'Athen', 'Vienna', 'Cairns',
'Berlin', 'Sydney'],
'inhabitants': [675218, 5000000, 664046, 1897000, 150041, 3769000, 5312000]})
# get value error
# how can this be solved differently?
cities_all.assign(in_cities_specific=np.where(cities_specific.city.isin(cities_all.city), '1', '0'))
# that's the solution I would like to get
expected_solution = pd.DataFrame({'city': ['Vancouver', 'Melbourne', 'Athen', 'Vienna', 'Cairns',
'Berlin', 'Sydney'],
'inhabitants': [675218, 5000000, 664046, 1897000, 150041, 3769000, 5312000],
'in_cities': [0, 1, 0, 0, 1, 0, 1]})

I think you are changing the position in the condition.
Here you have some alternatives:
cities_all.assign(
in_cities_specific=np.where(cities_all.city.isin(cities_specific.city), '1', '0')
)
or
cities_all["in_cities_specific"] =
cities_all["city"].isin(cities_specific["city"]).astype(int).astype(str)
or
condlist = [cities_all["city"].isin(cities_specific["city"])]
choicelist = ["1"]
cities_all["in_cities_specific"] = np.select(condlist, choicelist,default="0")

Print beauty dataframe in Jupyter Notebook

I would like to beautify my print output for a dataframe.
d = {'Carid ': [1, 2, 3], 'Carname': ['Mercedes-Benz', 'Audi', 'BMW'], 'model': ['S-Klasse AMG 63s', 'S6', 'X6 M-Power']}
df = pd.DataFrame(data=d)
print(df.head())
df.head()
As you can see the print-outpot not beauty. The last statement of df.head() is beauty.
Is there any option to get the same result in the print-statement in Jupyter Notebook?

Use display instead of print.
display(df.head())

Sure there is. You'll want to use Jupyter's built-in display, which you'll need to import first:
from IPython.display import display
Then, instead of print, use display:
display(df.head())
Source: https://tdhopper.com/blog/printing-pandas-data-frames-as-html-in-jupyter-notebooks