Let's say I have table which would look like that
| id | value_one | type | value_two |
|----|-----------|------|-----------|
| 1 | 2 | A | 1 |
| 1 | 4 | B | 1 |
| 2 | 3 | A | 2 |
| 2 | 1 | B | 3 |
I know that there are only A and B types for specific ID, what I want to achieve is to group those two values and calculate new type using formula A/B, it should be applied to value_one and value_two, so table afterwards should look like:
| id | value_one | type | value_two|
|----|-----------| -----|----------|
| 1 | 0.5 | C | 1 |
| 2 | 3 | C | 0.66 |
I am new to PySpark, and as for now I wasn't able to achieve described result, would appreciate any tips/solutions.
You can consider dividing the original dataframe into two parts according to type, and then use SQL statements to implement the calculation logic.
df.filter('type = "A"').createOrReplaceTempView('tmp1')
df.filter('type = "B"').createOrReplaceTempView('tmp2')
sql = """
select
tmp1.id
,tmp1.value_one / tmp2.value_one as value_one
,'C' as type
,tmp1.value_two / tmp2.value_two as value_two
from tmp1 join tmp2 using (id)
"""
reuslt_df = spark.sql(sql)
reuslt_df.show(truncate=False)
I have a dataframe X, where each row is a data point in time and each column is a feature. The label/target variable Y is univariate. One of the columns of X is the lagged values of Y.
The RNN input is of the shape (batch_size, n_timesteps, n_feature).
From what I've been reading on this site, batch_size should be as big as possible without running out of memory. My main doubt is about n_timesteps. and n_features.
I think n_feature is the number of columns in the X dataframe.
What about the n_timesteps?
Consider the following dataframe with the features temperature, pressure, and humidity:
import pandas as pd
import numpy as np
X = pd.DataFrame(data={
'temperature': np.random.random((1, 20)).ravel(),
'pressure': np.random.random((1, 20)).ravel(),
'humidity': np.random.random((1, 20)).ravel(),
})
print(X.to_markdown())
| | temperature | pressure | humidity |
|---:|--------------:|-----------:|-----------:|
| 0 | 0.205905 | 0.0824903 | 0.629692 |
| 1 | 0.280732 | 0.107473 | 0.588672 |
| 2 | 0.0113955 | 0.746447 | 0.156373 |
| 3 | 0.205553 | 0.957509 | 0.184099 |
| 4 | 0.741808 | 0.689842 | 0.0891679 |
| 5 | 0.408923 | 0.0685223 | 0.317061 |
| 6 | 0.678908 | 0.064342 | 0.219736 |
| 7 | 0.600087 | 0.369806 | 0.632653 |
| 8 | 0.944992 | 0.552085 | 0.31689 |
| 9 | 0.183584 | 0.102664 | 0.545828 |
| 10 | 0.391229 | 0.839631 | 0.00644447 |
| 11 | 0.317618 | 0.288042 | 0.796232 |
| 12 | 0.789993 | 0.938448 | 0.568106 |
| 13 | 0.0615843 | 0.704498 | 0.0554465 |
| 14 | 0.172264 | 0.615129 | 0.633329 |
| 15 | 0.162544 | 0.439882 | 0.0185174 |
| 16 | 0.48592 | 0.280436 | 0.550733 |
| 17 | 0.0370098 | 0.790943 | 0.592646 |
| 18 | 0.371475 | 0.976977 | 0.460522 |
| 19 | 0.493215 | 0.381539 | 0.995716 |
Now, if you want to use this kind of data for time series prediction with a RNN model, you usually consider one row in the data frame as one timestep. Converting the dataframe into an array might also help you understand what the timesteps are:
print(np.expand_dims(X.to_numpy(), axis=1).shape)
# (20, 1, 3)
First, I obtain an array of the shape(20, 3) or in other words, 20 samples and each sample has three features. I then explicitly add a time dimension to the array, resulting in the shape(20, 1, 3), meaning that the data set consists of 20 samples and each sample has one time step and for each time step you have 3 features. Now, you can use this data directly as input for a RNN.
I have some data stored in my pandas dataframe that shows salary for a bunch of users and their category.
| category | user_id | salary |
|----------|-----------|--------|
| A | 546457568 | 49203 |
| C | 356835679 | 49694 |
| A | 356785637 | 48766 |
| B | 45668758 | 36627 |
| C | 686794 | 59508 |
| C | 234232376 | 32765 |
| C | 4356345 | 44058 |
| A | 9878987 | 9999999|
What i would like to do is generate a new column salary_bucket that shows a bucket for salary, that is determined from the upper/lower limits of the Interquartile range for salary.
e.g. calculate upper/lower limits according to q1 - 1.5 x iqr and q3 + 1.5 x iqr, then split this into 10 equal buckets and assign each row to the relevant bucket based on salary. I know from exploration that there is no data outside the lower limit , but for data above the upper limit I would like a seperate bucket such as outside_iqr.
In the end I would liek to get something like so:
| category | user_id | salary | salary_bucket |
|----------|-----------|--------|---------------|
| A | 546457568 | 49203 | 7 |
| C | 356835679 | 49694 | 7 |
| A | 356785637 | 48766 | 7 |
| B | 45668758 | 36627 | 3 |
| C | 686794 | 59508 | 5 |
| C | 234232376 | 32765 | 3 |
| C | 4356345 | 44058 | 4 |
| A | 9878987 | 9999999|outside_iqr |
(these buckets are not actually calculate just for illustration sake)
Is something like qcut useful here?
You can use pandas.cut to turn continuous data into categorical data.
# First, we need to calculate our IQR.
q1 = df.salary.quantile(0.25)
q3 = df.salary.quantile(0.75)
iqr = q3 - q1
# Now let's calculate upper and lower bounds.
lower = q1 - 1.5*iqr
upper = q3 + 1.5*iqr
# Let us create our bins:
num_bins = 10
bin_width = (upper - lower) / num_bins
bins = [lower + i*bin_width for i in range(num_bins)]
bins += [upper, float('inf')] # Now we add our last bin, which will contain any value greater than the upper-bound of the IQR.
# Let us create our labels:
labels = [f'Bucket {i}' for i in range(1,num_bins+1)]
labels.append('Outside IQR')
# Finally, we add a new column to the df:
df['salary_bucket'] = pd.cut(df.salary, bins=bins, labels=labels)
So basically, you'll need to generate your own list of buckets and labels according to what you require, and then pass those as arguments to pandas.cut.
I have some data as follows:
+--------+------+
| Reason | Keys |
+--------+------+
| x | a |
| y | a |
| z | a |
| y | b |
| z | b |
| x | c |
| w | d |
| x | d |
| w | d |
+--------+------+
I want to get the Reason corresponding to the first occurrence of each Key. Like here, I should get Reasons x,y,x,w for Keys a,b,c,d respectively. After that, I want to compute the percentage of each Reason, as in a metric for how many times each Reason occurs. Thus x = 2/4 = 50%. And w,y = 25% each.
For the percentage, I think I can use something like value_counts(normalize=True) * 100, based on the previous step. What is a good way to proceed?
You are right about the second step and the first step could be achieved by
summary = df.groupby("Keys").first()
You can using drop_duplicates
df.drop_duplicates(['Reason'])
Out[207]:
Reason Keys
0 x a
1 y a
2 z a
6 w d
I am using the Agate library to create a table.
Using the command as :
table = agate.Table(cpi_rows, cpi_types, cpi_titles)
Sample values are as below :
cpi_rows[0]
[1.0,'Denmark','DNK',128.0,'EU',1.0,91.0,7.0,2.2,87.0,95.0,83.0,98.0,0.0,97.0,0.0,96.0,98.0,0.0,87.0,89.0,88.0,83.0,0.0,0.0,0.0]
cpi_tiles
['Country Rank','Country / Territory','WB Code','IFS Code','Region','Country Rank','CPI 2013 Score', 'Surveys Used','Standard Error', '90% Confidence interval Lower', 'Upper','Scores range MIN','MAX','Data sources AFDB','BF (SGI)','BF (BTI)','IMD','ICRG','WB','WEF','WJP','EIU','GI','PERC','TI','FH']
When I run the command, I am getting the error as :
ValueError: Column names must be strings or None.
Though all the names in cpi_titles are type strings only, I am unable to get the cause for error.
Just tried your code, and apart from a few corrections to names and stuff this worked without a problem
cpi_rows = [[]]
cpi_rows[0] =[1.0,'Denmark','DNK',128.0,'EU',1.0,91.0,7.0,2.2,87.0,95.0,83.0,98.0,0.0,97.0,0.0,96.0,98.0,0.0,87.0,89.0,88.0,83.0,0.0,0.0,0.0]
cpi_titles = ['Country Rank','Country / Territory','WB Code','IFS Code','Region','Country Rank','CPI 2013 Score', 'Surveys Used','Standard Error', '90% Confidence interval Lower', 'Upper','Scores range MIN','MAX','Data sources AFDB','BF (SGI)','BF (BTI)','IMD','ICRG','WB','WEF','WJP','EIU','GI','PERC','TI','FH']
table = agate.Table(cpi_rows, cpi_titles)
print table.print_structure()
The output is
| column | data_type |
| ----------------------------- | --------- |
| Country Rank | Number |
| Country / Territory | Text |
| WB Code | Text |
| IFS Code | Number |
| Region | Text |
| Country Rank_2 | Number |
| CPI 2013 Score | Number |
| Surveys Used | Number |
| Standard Error | Number |
| 90% Confidence interval Lower | Number |
| Upper | Number |
| Scores range MIN | Number |
| MAX | Number |
| Data sources AFDB | Number |
| BF (SGI) | Number |
| BF (BTI) | Number |
| IMD | Number |
| ICRG | Number |
| WB | Number |
| WEF | Number |
| WJP | Number |
| EIU | Number |
| GI | Number |
| PERC | Number |
| TI | Number |
| FH | Number |
Obviously, I don't have your definition of types which you want to apply to this data. The only other thing to note is that you have defined Country Rank twice in your column titles so Agate does warn you about this and relabel it.