Currently, I have managed to solve this but it is slower than what I need. It takes approximately: 1 hour for 500k samples, the entire dataset is ~100M samples, which requires ~200hours for 100M samples.
Hardware/Software specs: RAM 8GB, Windows 11 64bit, Python 3.8.8
The problem:
I have a dataset in .csv (~13GB) where each sample has a value and a respective start-end period of few months.I want to create a dataset where each sample will have the same value but referring to each specific month.
For example:
from:
idx | start date | end date | month | year | value
0 | 20/05/2022 | 20/07/2022 | 0 | 0 | X
to:
0 | 20/05/2022 | 20/07/2022 | 5 | 2022 | X
1 | 20/05/2022 | 20/07/2022 | 6 | 2022 | X
2 | 20/05/2022 | 20/07/2022 | 7 | 2022 | X
Ideas: Manage to do it parallel (like Dask, but I am not sure how for this task).
My implementation:
Chunk read in pandas, augment in dictionaries , append to CSV. Use a function that, given a df, calculates for each sample the months from start date to end date and creates a copy sample for each month appending it to a dictionary. Then it returns the final dictionary.
The calculations are done in dictionaries as they were found to be way faster than doing it in pandas. Then I iterate through the original CSV in chunks and apply the function at each chunk appending the resulting augmented df to another csv.
The function:
def augment_to_monthly_dict(chunk):
'''
Function takes a df or subdf data and creates and returns an Augmented dataset with monthly data in
Dictionary form (for efficiency)
'''
dict={}
l=1
for i in range(len(chunk)):#iterate through every sample
# print(str(chunk.iloc[i].APO)[4:6] )
#Find the months and years period
mst =int(float((str(chunk.iloc[i].start)[4:6])))#start month
mend=int(str(chunk.iloc[i].end)[4:6]) #end month
yst =int(str(chunk.iloc[i].start)[:4] )#start year
yend=int(str(chunk.iloc[i].end)[:4] )#end year
if yend==yst:
months=[ m for m in range(mst,mend+1)]
years=[yend for i in range(len(months))]
elif yend==yst+1:# year change at same sample
months=[m for m in range(mst,13)]
years=[yst for i in range(mst,13)]
months= months+[m for m in range(1, mend+1)]
years= years+[yend for i in range(1, mend+1)]
else:
continue
#months is a list of each month in the period of the sample and years is a same
#length list of the respective years eg months=[11,12,1,2] , years=
#[2021,2022,2022,2022]
for j in range(len(months)):#iterate through list of months
#copy the original sample make it a dictionary
tmp=pd.DataFrame(chunk.iloc[i]).transpose().to_dict(orient='records')
#change the month and year values accordingly (they were 0 for initiation)
tmp[0]['month'] = months[j]
tmp[0]['year'] = years[j]
# Here could add more calcs e.g. drop irrelevant columns, change datatypes etc
#to reduce size
#
#-------------------------------------
#Append new row to the Augmented data
dict[l] = tmp[0]
l+=1
return dict
Reading the original dataset (.csv ~13GB), augment using the function and append result to new .csv:
chunk_count=0
for chunk in pd.read_csv('enc_star_logar_ek.csv', delimiter=';', chunksize=10000):
chunk.index = chunk.reset_index().index
aug_dict = augment_to_monthly_dict(chunk)#make chunk dictionary to work faster
chunk_count+=1
if chunk_count ==1: #get the column names and open csv write headers and 1st chunk
#Find the dicts keys, the column names only from the first dict(not reading all data)
for kk in aug_dict.values():
key_names = [i for i in kk.keys()]
print(key_names)
break #break after first input dict
#Open csv file and write ';' separated data
with open('dic_to_csv2.csv', 'w', newline='') as csvfile:
writer = csv.DictWriter(csvfile,delimiter=';', fieldnames=key_names)
writer.writeheader()
writer.writerows(aug_dict.values())
else: # Save the rest of the data chunks
print('added chunk: ', chunk_count)
with open('dic_to_csv2.csv', 'a', newline='') as csvfile:
writer = csv.DictWriter(csvfile,delimiter=';', fieldnames=key_names)
writer.writerows(aug_dict.values())
Pandas efficiency comes in to play when you need to manipulate columns of data, and to do that Pandas reads the input row-by-row building up a series of data for each column; that's a lot of extra computation your problem doesn't benefit from, and in fact just slows your solution down.
You actually need to manipulate rows, and for that the fastest way is to use the standard csv module; all you need to do is read a row in, write the derived rows out, and repeat:
import csv
import sys
from datetime import datetime
def parse_dt(s):
return datetime.strptime(s, r"%d/%m/%Y")
def get_dt_range(beg_dt, end_dt):
"""
Returns a range of (month, year) tuples, from beg_dt up-to-and-including end_dt.
"""
if end_dt < beg_dt:
raise ValueError(f"end {end_dt} is before beg {beg_dt}")
mo, yr = beg_dt.month, beg_dt.year
dt_range = []
while True:
dt_range.append((mo, yr))
if mo == 12:
mo = 1
yr = yr + 1
else:
mo += 1
if (yr, mo) > (end_dt.year, end_dt.month):
break
return dt_range
fname = sys.argv[1]
with open(fname, newline="") as f_in, open("output_csv.csv", "w", newline="") as f_out:
reader = csv.reader(f_in)
writer = csv.writer(f_out)
writer.writerow(next(reader)) # transfer header
for row in reader:
beg_dt = parse_dt(row[1])
end_dt = parse_dt(row[2])
for mo, yr in get_dt_range(beg_dt, end_dt):
row[3] = mo
row[4] = yr
writer.writerow(row)
And, to compare with Pandas in general, let's examine #abokey's specifc Pandas solution—I'm not sure if there is a better Pandas implementation, but this one kinda does the right thing:
import sys
import pandas as pd
fname = sys.argv[1]
df = pd.read_csv(fname)
df["start date"] = pd.to_datetime(df["start date"], format="%d/%m/%Y")
df["end date"] = pd.to_datetime(df["end date"], format="%d/%m/%Y")
df["month"] = df.apply(
lambda x: pd.date_range(
start=x["start date"], end=x["end date"] + pd.DateOffset(months=1), freq="M"
).month.tolist(),
axis=1,
)
df["year"] = df["start date"].dt.year
out = df.explode("month").reset_index(drop=True)
out.to_csv("output_pd.csv")
Let's start with the basics, though, do the programs actually do the right thing. Given this input:
idx,start date,end date,month,year,value
0,20/05/2022,20/05/2022,0,0,X
0,20/05/2022,20/07/2022,0,0,X
0,20/12/2022,20/01/2023,0,0,X
My program, ./main.py input.csv, produces:
idx,start date,end date,month,year,value
0,20/05/2022,20/05/2022,5,2022,X
0,20/05/2022,20/07/2022,5,2022,X
0,20/05/2022,20/07/2022,6,2022,X
0,20/05/2022,20/07/2022,7,2022,X
0,20/12/2022,20/01/2023,12,2022,X
0,20/12/2022,20/01/2023,1,2023,X
I believe that's what you're looking for.
The Pandas solution, ./main_pd.py input.csv, produces:
,idx,start date,end date,month,year,value
0,0,2022-05-20,2022-05-20,5,2022,X
1,0,2022-05-20,2022-07-20,5,2022,X
2,0,2022-05-20,2022-07-20,6,2022,X
3,0,2022-05-20,2022-07-20,7,2022,X
4,0,2022-12-20,2023-01-20,12,2022,X
5,0,2022-12-20,2023-01-20,1,2022,X
Ignoring the added column for the frame index, and the fact the date format has been changed (I'm pretty sure that can be fixed with some Pandas directive I don't know), it still does the right thing with regards to creating new rows with the appropriate date range.
So, both do the right thing. Now, on to performance. I duplicated your initial sample, just the 1 row, for 1_000_000 and 10_000_000 rows:
import sys
nrows = int(sys.argv[1])
with open(f"input_{nrows}.csv", "w") as f:
f.write("idx,start date,end date,month,year,value\n")
for _ in range(nrows):
f.write("0,20/05/2022,20/07/2022,0,0,X\n")
I'm running a 2020, M1 MacBook Air with the 2TB SSD (which gives very good read/write speeds):
1M rows (sec, RAM)
10M rows (sec, RAM)
csv module
7.8s, 6MB
78s, 6MB
Pandas
75s, 569MB
750s, 5.8GB
You can see both programs following a linear increase in time-to-run that follows the increase in the size of rows. The csv module's memory remains constanly non-existent because it's streaming data in-and-out (holding on to virtually nothing); Pandas's memory rises with the size of rows it has to hold so that it can do the actual date-range computations, again on whole columns. Also, not shown, but for the 10M-rows Pandas test, Pandas spent nearly 2 minutes just writing the CSV—longer than the csv-module approach took to complete the entire task.
Now, for all my putting-down of Pandas, the solution is far fewer lines, and is probably bug free from the get-go. I did have a problem writing get_dt_range(), and had to spend about 5 minutes thinking about what it actually needed to do and debug it.
You can view my setup with the small test harness, and the results, here.
I suggest you to use pandas (or even dask) to return the list of months between two columns of a huge dataset (e.g, .csv ~13GB). First you need to convert your two columns to a datetime by using pandas.to_datetime. Then, you can use pandas.date_range to get your list.
Try with this :
import pandas as pd
from io import StringIO
s = """start date end date month year value
20/05/2022 20/07/2022 0 0 X
"""
df = pd.read_csv(StringIO(s), sep='\t')
df['start date'] = pd.to_datetime(df['start date'], format = "%d/%m/%Y")
df['end date'] = pd.to_datetime(df['end date'], format = "%d/%m/%Y")
df["month"] = df.apply(lambda x: pd.date_range(start=x["start date"], end=x["end date"] + pd.DateOffset(months=1), freq="M").month.tolist(), axis=1)
df['year'] = df['start date'].dt.year
out = df.explode('month').reset_index(drop=True)
>>> print(out)
start date end date month year value
0 2022-05-20 2022-07-20 5 2022 X
1 2022-05-20 2022-07-20 6 2022 X
2 2022-05-20 2022-07-20 7 2022 X
Note : I tested the code above on a 1 million .csv dataset and it took ~10min to get the output.
you can read very large csv file with dask, then process it (same api as pandas), then convert it to pandas dataframe if you need.
dask is perfect when pandas fails due to data size or computation speed. But for data that fits into RAM, pandas can often be faster and easier to use than Dask DataFrame.
import dask.dataframe as dd
#1. read the large csv
dff = dd.read_csv('path_to_big_csv_file.csv') #return Dask.DataFrame
#if still not enough, try more reducing IO costs:
dff = dd.read_csv('largefile.csv', blocksize=25e6) #use blocksize (number of bytes by which to cut up larger files)
dff = dd.read_csv('largefile.csv', columns=["a", "b", "c"]) #return only columns a, b and c
#2. work with dff, dask has the same api than pandas:
#https://docs.dask.org/en/stable/dataframe-api.html
#3. then, finally, convert dff to pandas dataframe if you want
df = dff.compute() #return pandas dataframe
you can also try other alternatives for reading very large csv files efficiently with high speed & low momory usage:
pola, modin, koalas.
all those packages, same as dask, use similar api as pandas.
if you have very big csv file, pandas read_csv with chunksize usually don't succeed, and even if if succeed, it will be waist of time and energy
There's a Table helper in convtools library (I must confess, a lib of mine). This helper processes csv files as a stream, using simple csv.reader under the hood:
from datetime import datetime
from convtools import conversion as c
from convtools.contrib.tables import Table
def dt_range_to_months(dt_start, dt_end):
return tuple(
(year_month // 12, year_month % 12 + 1)
for year_month in range(
dt_start.year * 12 + dt_start.month - 1,
dt_end.year * 12 + dt_end.month,
)
)
(
Table.from_csv("tmp/in.csv", header=True)
.update(
year_month=c.call_func(
dt_range_to_months,
c.call_func(datetime.strptime, c.col("start date"), "%d/%m/%Y"),
c.call_func(datetime.strptime, c.col("end date"), "%d/%m/%Y"),
)
)
.explode("year_month")
.update(
year=c.col("year_month").item(0),
month=c.col("year_month").item(1),
)
.drop("year_month")
.into_csv("tmp/out.csv")
)
Input/Output:
~/o/convtools ❯❯❯ head tmp/in.csv
idx,start date,end date,month,year,value
0,20/05/2022,20/07/2022,0,0,X
0,20/05/2022,20/07/2022,0,0,X
0,20/05/2022,20/07/2022,0,0,X
0,20/05/2022,20/07/2022,0,0,X
0,20/05/2022,20/07/2022,0,0,X
0,20/05/2022,20/07/2022,0,0,X
0,20/05/2022,20/07/2022,0,0,X
0,20/05/2022,20/07/2022,0,0,X
0,20/05/2022,20/07/2022,0,0,X
...
~/o/convtools ❯❯❯ head tmp/out.csv
idx,start date,end date,month,year,value
0,20/05/2022,20/07/2022,5,2022,X
0,20/05/2022,20/07/2022,6,2022,X
0,20/05/2022,20/07/2022,7,2022,X
0,20/05/2022,20/07/2022,5,2022,X
0,20/05/2022,20/07/2022,6,2022,X
0,20/05/2022,20/07/2022,7,2022,X
0,20/05/2022,20/07/2022,5,2022,X
0,20/05/2022,20/07/2022,6,2022,X
0,20/05/2022,20/07/2022,7,2022,X
...
On my M1 Mac on a file where each row explodes into three it processes 100K of rows per second. In case of 100M rows of the same structure it should take ~ 1000s (< 17 min). Of course it depends on how deep inner by-month cycles are.
Only basic knowledge of Python, so I'm not even sure if this is possible?
I have a csv that looks like this:
[1]: https://i.stack.imgur.com/8clYM.png
(This is dummy data, the real one is about 30K rows.)
I need to find the most recent job title for each employee (unique id) and then calculate how long (= how many days) the employee has been on the same job title.
What I have done so far:
import csv
import datetime
from datetime import *
data = open("C:\\Users\\User\\PycharmProjects\\pythonProject\\jts.csv",encoding="utf-8")
csv_data = csv.reader(data)
data_lines = list(csv_data)
print(data_lines)
for i in data_lines:
for j in i[0]:
But then I haven't got anywhere because I can't even conceptualise how to structure this. :-(
I also know that at one point I will need:
datetime.strptime(data_lines[1][2] , '%Y/%M/%d').date()
Could somebody help, please? I just need a new list saying something like:
id jt days
500 plumber 370
Edit to clarify: The dates are data points taken. I need to calculate back from the most recent of those back until the job title was something else. So in my example for employee 5000 from 04/07/2021 to 01/03/2020.
Let's consider sample data as follows:
id,jtitle,date
5000,plumber,01/01/2020
5000,senior plumber,02/03/2020
6000,software engineer,01/02/2020
6000,software architecture,06/02/2021
7000,software tester,06/02/2019
The following code works.
import pandas as pd
import datetime
# load data
data = pd.read_csv('data.csv')
# convert to datetime object
data.date = pd.to_datetime(data.date, dayfirst=True)
print(data)
# group employees by ID
latest = data.sort_values('date', ascending=False).groupby('id').nth(0)
print(latest)
# find the latest point in time where there is a change in job title
prev_date = data.sort_values('date', ascending=False).groupby('id').nth(1).date
print(prev_date)
# calculate the difference in days
latest['days'] = latest.date - prev_date
print(latest)
Output:
jtitle date days
id
5000 senior plumber 2020-03-02 61 days
6000 software architecture 2021-02-06 371 days
7000 software tester 2019-02-06 NaT
But then I haven't got anywhere because I can't even conceptualise how to structure this. :-(
Have a map (dict) of employee to (date, title).
For every row, check if you already have an entry for the employee. If you don't just put the information in the map, otherwise compare the date of the row and that of the entry. If the row has a more recent date, replace the entry.
Once you've gone through all the rows, you can just go through the map you've collected and compute the difference between the date you ended up with and "today".
Incidentally your pattern is not correct, the sample data uses a %d/%m/%Y (day/month/year) or %m/%d/%Y (month/day/year) format, the sample data is not sufficient to say which, but it certainly is not YMD.
Seems like I'm too late... Nevertheless, in case you're interested, here's a suggestion in pure Python (nothing wrong with Pandas, though!):
import csv
import datetime as dt
from operator import itemgetter
from itertools import groupby
reader = csv.reader('data.csv')
next(reader) # Discard header row
# Read, transform (date), and sort in reverse (id first, then date):
data = sorted(((i, jtitle, dt.datetime.strptime(date, '%d/%m/%Y'))
for i, jtitle, date in reader),
key=itemgetter(0, 2), reverse=True)
# Process data grouped by id
result = []
for i, group in groupby(data, key=itemgetter(0)):
_, jtitle, end = next(group) # Fetch last job title resp. date
# Search for first ocurrence of different job title:
start = end
for _, jt, start in group:
if jt != jtitle:
break
# Collect results in list with datetimes transformed back
result.append((i, jtitle, end.strftime('%d/%m/%Y'), (end - start).days))
result = sorted(result, key=itemgetter(0))
The result for the input data
id,jtitle,date
5000,plumber,01/01/2020
5000,plumber,01/02/2020
5000,senior plumber,01/03/2020
5000,head plumber,01/05/2020
5000,head plumber,02/09/2020
5000,head plumber,05/01/2021
5000,head plumber,04/07/2021
6000,electrician,01/02/2018
6000,qualified electrician,01/06/2020
7000,plumber,01/01/2004
7000,plumber,09/11/2020
7000,senior plumber,05/06/2021
is
[('5000', 'head plumber', '04/07/2021', 490),
('6000', 'qualified electrician', '01/06/2020', 851),
('7000', 'senior plumber', '05/06/2021', 208)]
I am currently working on a Python project, that imports a data text file (CSV in my case) and then outputs the employees who have worked the most time together in a common project. First, this is the code and the data file:
from collections import defaultdict
from itertools import combinations
from datetime import datetime
import csv
d = defaultdict(list)
with open("data.csv") as f:
next(f) # skip header
r = csv.reader(f)
# unpack use height as key and append name age and position
for EmpID, ProjectID, FromDate, ToDate in r:
d[int(ProjectID)].append((EmpID, FromDate, ToDate))
for job, aref in d.items():
if len(aref) >= 2:
for ref in combinations(aref, 2):
begin = max(map(lambda x: x[1], ref))
end = min(map(lambda x: x[2], ref))
delta = datetime.strptime(end, '%Y-%m-%d') \
- datetime.strptime(begin, '%Y-%m-%d')
dd = delta.days
if dd > 0:
print('Employees with EmpID:', ref[0][0], 'and', ref[1][0],
'worked together on a common project (Project ID:', job, ') for a total of', dd, 'days')
And this is the data file, I am importing:
EmpID,ProjectID,DateFrom,DateTo
1,100,2014-11-01,2015-05-01
2,101,2013-12-06,2014-10-06
3,102,2015-06-04,2017-09-04
5,103,2014-10-01,2015-12-01
2,100,2013-03-07,2015-11-07
2,103,2015-07-09,2019-01-19
4,102,2013-11-13,2014-03-13
4,103,2016-02-14,2017-03-15
5,104,2014-03-15,2015-11-09
Now, I have a task that if there is a value 'NULL' in the 'DateTo' column, I have to make it equal today. I am thinking that there should be an automatic python function that gives the current date, and then do an if statement inside the CSV code block to replace 'NULL' with today's date (but it is only open in read mode as far as I know?). I would very much appreciate it if anyone can give me any tips! Thanks.
EDIT:
PANDAS PREVIOUS ATTEMPT FOR SOLUTION: (50% done)
# Load the Pandas libraries with alias 'pd'
import pandas as pd
import datetime as dt
import numpy as np
# Read data from file 'filename.csv'
# (in the same directory that your python process is based)
# Control delimiters, rows, column names with read_csv (see later)
date_parser = lambda c: pd.to_datetime(c, format='%Y/%m/%d', errors='coerce')
df = pd.read_csv('data.csv', delimiter = ',', parse_dates=[2,3], date_parser=date_parser)
df.set_index("EmpID", inplace = True)
df.sort_values(['ProjectID'], inplace=True)
df['Days Worked'] = (df['DateTo'] - df['DateFrom']).dt.days
cutdown_projecs = df.groupby('ProjectID').filter(lambda x: len(x) >= 2)
print(cutdown_projecs)
use fillna
import pandas as pd
from io import StringIO
from datetime import datetime
document = '''
EmpID,ProjectID,DateFrom,DateTo
1,100,2014-11-01,2015-05-01
2,101,2013-12-06,2014-10-06
3,102,2015-06-04,2017-09-04
5,103,2014-10-01,2015-12-01
2,100,2013-03-07,NULL
2,103,2015-07-09,2019-01-19
4,102,2013-11-13,2014-03-13
4,103,2016-02-14,2017-03-15
5,104,2014-03-15,2015-11-09'''
# df = pd.read_csv('data.csv')
df = pd.read_csv(StringIO(document))
df['DateTo'] = df['DateTo'].fillna(datetime.today().strftime('%Y-%m-%d'))
print(df)
EmpID ProjectID DateFrom DateTo
0 1 100 2014-11-01 2015-05-01
1 2 101 2013-12-06 2014-10-06
2 3 102 2015-06-04 2017-09-04
3 5 103 2014-10-01 2015-12-01
4 2 100 2013-03-07 2019-05-30
5 2 103 2015-07-09 2019-01-19
6 4 102 2013-11-13 2014-03-13
7 4 103 2016-02-14 2017-03-15
8 5 104 2014-03-15 2015-11-09
In your code is possible use if-else with condition for test empty values string values and replace by today datetime without times:
import pandas as pd
end = min(map(lambda x: x[2], ref))
end = datetime.strptime(end, '%Y-%m-%d') if end != '' else pd.Timestamp("today").floor('d')
delta = end - datetime.strptime(begin, '%Y-%m-%d')
Similar to this question, but my CSV has a slightly different format. Here is an example:
id,employee,details,createdAt
1,John,"{"Country":"USA","Salary":5000,"Review":null}","2018-09-01"
2,Sarah,"{"Country":"Australia", "Salary":6000,"Review":"Hardworking"}","2018-09-05"
I think the double quotation mark in the beginning of the JSON column might have caused some errors. Using df = pandas.read_csv('file.csv'), this is the dataframe that I got:
id employee details createdAt Unnamed: 1 Unnamed: 2
1 John {Country":"USA" Salary:5000 Review:null}" 2018-09-01
2 Sarah {Country":"Australia" Salary:6000 Review:"Hardworking"}" 2018-09-05
My desired output:
id employee details createdAt
1 John {"Country":"USA","Salary":5000,"Review":null} 2018-09-01
2 Sarah {"Country":"Australia","Salary":6000,"Review":"Hardworking"} 2018-09-05
I've tried adding quotechar='"' as the parameter and it still doesn't give me the result that I want. Is there a way to tell pandas to ignore the first and the last quotation mark surrounding the json value?
As an alternative approach you could read the file in manually, parse each row correctly and use the resulting data to contruct the dataframe. This works by splitting the row both forward and backwards to get the non-problematic columns and then taking the remaining part:
import pandas as pd
data = []
with open("e1.csv") as f_input:
for row in f_input:
row = row.strip()
split = row.split(',', 2)
rsplit = [cell.strip('"') for cell in split[-1].rsplit(',', 1)]
data.append(split[0:2] + rsplit)
df = pd.DataFrame(data[1:], columns=data[0])
print(df)
This would display your data as:
id employee details createdAt
0 1 John {"Country":"USA","Salary":5000,"Review":null} 2018-09-01
1 2 Sarah {"Country":"Australia", "Salary":6000,"Review"... 2018-09-05
I have reproduced your file
With
df = pd.read_csv('e1.csv', index_col=None )
print (df)
Output
id emp details createdat
0 1 john "{"Country":"USA","Salary":5000,"Review":null}" "2018-09-01"
1 2 sarah "{"Country":"Australia", "Salary":6000,"Review... "2018-09-05"
I think there's a better way by passing a regex to sep=r',"|",|(?<=\d),' and possibly some other combination of parameters. I haven't figured it out totally.
Here is a less than optimal option:
df = pd.read_csv('s083838383.csv', sep='##$%^', engine='python')
header = df.columns[0]
print(df)
Why sep='##$%^' ? This is just garbage that allows you to read the file with no sep character. It could be any random character and is just used as a means to import the data into a df object to work with.
df looks like this:
id,employee,details,createdAt
0 1,John,"{"Country":"USA","Salary":5000,"Review...
1 2,Sarah,"{"Country":"Australia", "Salary":6000...
Then you could use str.extract to apply regex and expand the columns:
result = df[header].str.extract(r'(.+),(.+),("\{.+\}"),(.+)',
expand=True).applymap(str.strip)
result.columns = header.strip().split(',')
print(result)
result is:
id employee details createdAt
0 1 John "{"Country":"USA","Salary":5000,"Review":null}" "2018-09-01"
1 2 Sarah "{"Country":"Australia", "Salary":6000,"Review... "2018-09-05"
If you need the starting and ending quotes stripped off of the details string values, you could do:
result['details'] = result['details'].str.strip('"')
If the details object items needs to be a dicts instead of strings, you could do:
from json import loads
result['details'] = result['details'].apply(loads)