Related
How do I use translate in tensorflow_addons.image?
I tried the following with no success
(Versions: tensorflow: 2.4.1, tensorflow-addons: 0.12.1, python: 3.8.8)
import tensorflow as tf
import numpy as np
from tensorflow_addons.image import translate as tfa_translate
# 5 images 10x10, BW:
imgs = np.random.rand(5, 10, 10, 1).astype(np.float32)
# vector for x-y translations:
vec = np.random.randn(5, 2).astype(np.float32)
# both of these give me the same error below:
tfa_translate(imgs, vec)
tfa_translate(tf.convert_to_tensor(imgs), tf.convert_to_tensor(vec))
The error I get is about using tf or np tensors. I tried with tf.convert_to_tensor(imgs) but no success:
NotImplementedError: Cannot convert a symbolic Tensor
(translate/translations_to_projective_transforms/strided_slice:0) to a numpy array.
This error` may indicate that you're trying to pass a Tensor to a NumPy call,
which is not supported
Edit: All the ops on images in these examples work in my systems. But translate does not.
I was able to execute above code in this combination tensorflow: 2.4.1, tensorflow-addons: 0.12.1 and python: 3.7.10. You will be success.
import sys
import tensorflow as tf
import numpy as np
import tensorflow_addons as tfa
from tensorflow_addons.image import translate as tfa_translate
print(tf.__version__)
print(tfa.__version__)
print(sys.version)
# 5 images 10x10, BW:
imgs = np.random.rand(5, 10, 10, 1).astype(np.float32)
# vector for x-y translations:
vec = np.random.randn(5, 2).astype(np.float32)
# both of these give me the same error below:
tfa_translate(imgs, vec)
tfa_translate(tf.convert_to_tensor(imgs), tf.convert_to_tensor(vec))
Output:
2.4.1
0.12.1
3.7.10
<tf.Tensor: shape=(5, 10, 10, 1), dtype=float32, numpy=
array([[[[0.73720914],
[0.05389478],
[0.00823868],
[0.81953126],
[0.1538573 ],
[0.90768087],
[0.4161973 ],
[0.38890013],
[0.7050291 ],
[0.20925364]],
[[0.37302 ],
[0.4195738 ],
[0.6881046 ],
[0.3580131 ],
[0.47772875],
[0.5959794 ],
[0.60560894],
[0.03321437],
[0.48405352],
[0.7554338 ]],
[[0.8335805 ],
[0.69475925],
[0.7356104 ],
[0.72107244],
[0.0654715 ],
[0.7524717 ],
[0.7242604 ],
[0.8744024 ],
[0.4255406 ],
[0.8113107 ]],
[[0.6129395 ],
[0.29516342],
[0.9028247 ],
[0.13342927],
[0.52949995],
[0.06142981],
[0.88151217],
[0.52935964],
[0.9567349 ],
[0.8202645 ]],
[[0.16741282],
[0.6734363 ],
[0.1131402 ],
[0.8026573 ],
[0.468824 ],
[0.8553658 ],
[0.45999104],
[0.6134152 ],
[0.56909144],
[0.41677374]],
[[0.35705274],
[0.76314545],
[0.07150301],
[0.487003 ],
[0.13564219],
[0.45595258],
[0.33955264],
[0.9714719 ],
[0.3500332 ],
[0.6937466 ]],
[[0.648089 ],
[0.6019247 ],
[0.3049946 ],
[0.20571095],
[0.2413046 ],
[0.03899029],
[0.85995656],
[0.73361784],
[0.83635384],
[0.7748581 ]],
[[0.4335603 ],
[0.5851778 ],
[0.11063848],
[0.80494994],
[0.49294096],
[0.25171277],
[0.81821585],
[0.85311455],
[0.6131411 ],
[0.7952116 ]],
[[0.7456568 ],
[0.6693832 ],
[0.05555756],
[0.11654849],
[0.674931 ],
[0.01180321],
[0.9755547 ],
[0.4340065 ],
[0.1364122 ],
[0.31032914]],
[[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ]]],
[[[0.34124446],
[0.08647306],
[0.9612431 ],
[0.8725844 ],
[0.6298909 ],
[0.55960226],
[0.3880017 ],
[0.15042423],
[0.58690244],
[0. ]],
[[0.81984246],
[0.8341217 ],
[0.35999405],
[0.3214622 ],
[0.27512822],
[0.905134 ],
[0.5411199 ],
[0.5287951 ],
[0.56175965],
[0. ]],
[[0.7129698 ],
[0.17935541],
[0.23184128],
[0.62122566],
[0.31914395],
[0.93782455],
[0.68015796],
[0.32218182],
[0.94558555],
[0. ]],
[[0.9850416 ],
[0.63286716],
[0.02245921],
[0.0168714 ],
[0.787904 ],
[0.7566337 ],
[0.6966046 ],
[0.74724126],
[0.12862223],
[0. ]],
[[0.9159414 ],
[0.44754454],
[0.4354201 ],
[0.00121895],
[0.9533174 ],
[0.6897298 ],
[0.5429657 ],
[0.89288723],
[0.17915004],
[0. ]],
[[0.35772014],
[0.25488794],
[0.7412978 ],
[0.77173555],
[0.44523025],
[0.515731 ],
[0.6853389 ],
[0.90855956],
[0.28565803],
[0. ]],
[[0.9447069 ],
[0.25103524],
[0.2835534 ],
[0.5845979 ],
[0.61925954],
[0.5675663 ],
[0.5097425 ],
[0.7441513 ],
[0.4252864 ],
[0. ]],
[[0.71359545],
[0.98984915],
[0.25157735],
[0.00311989],
[0.59827286],
[0.2282706 ],
[0.74741423],
[0.71861583],
[0.4377698 ],
[0. ]],
[[0.09316336],
[0.90708685],
[0.09424796],
[0.49117166],
[0.16317427],
[0.8758143 ],
[0.40267643],
[0.9177538 ],
[0.962702 ],
[0. ]],
[[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ]]],
[[[0. ],
[0. ],
[0.29238865],
[0.11071314],
[0.04251722],
[0.8379944 ],
[0.7078936 ],
[0.9524308 ],
[0.9807888 ],
[0.5980042 ]],
[[0. ],
[0. ],
[0.6076964 ],
[0.01608464],
[0.5086389 ],
[0.6162034 ],
[0.13877843],
[0.35256362],
[0.49832624],
[0.8281829 ]],
[[0. ],
[0. ],
[0.16112986],
[0.7567848 ],
[0.3147346 ],
[0.37325364],
[0.7240583 ],
[0.55212367],
[0.53437054],
[0.6456858 ]],
[[0. ],
[0. ],
[0.9337465 ],
[0.91347605],
[0.20626037],
[0.76980096],
[0.42335644],
[0.25908992],
[0.11779127],
[0.92509085]],
[[0. ],
[0. ],
[0.42465472],
[0.0791868 ],
[0.260981 ],
[0.98131573],
[0.75192803],
[0.94773096],
[0.56423026],
[0.41657394]],
[[0. ],
[0. ],
[0.06342959],
[0.14268413],
[0.62750655],
[0.40286288],
[0.5470112 ],
[0.16235335],
[0.5923443 ],
[0.2899487 ]],
[[0. ],
[0. ],
[0.46837795],
[0.64133584],
[0.19043817],
[0.3560451 ],
[0.84368664],
[0.54878306],
[0.7572676 ],
[0.63685185]],
[[0. ],
[0. ],
[0.6362246 ],
[0.9802078 ],
[0.07812112],
[0.6737748 ],
[0.09005726],
[0.5559371 ],
[0.8273103 ],
[0.8465655 ]],
[[0. ],
[0. ],
[0.29354632],
[0.8277752 ],
[0.24557658],
[0.7144999 ],
[0.14932555],
[0.7992488 ],
[0.9322711 ],
[0.7795503 ]],
[[0. ],
[0. ],
[0.6565971 ],
[0.9446715 ],
[0.30254883],
[0.87504804],
[0.64962775],
[0.13229366],
[0.3159616 ],
[0.80296475]]],
[[[0. ],
[0.70018524],
[0.7575729 ],
[0.9112962 ],
[0.7581045 ],
[0.48528266],
[0.7747405 ],
[0.24268304],
[0.51620704],
[0.34689027]],
[[0. ],
[0.95382404],
[0.80364156],
[0.48280716],
[0.37804484],
[0.24970461],
[0.9051866 ],
[0.7865404 ],
[0.06981565],
[0.44362214]],
[[0. ],
[0.2775535 ],
[0.9653021 ],
[0.63792914],
[0.7292578 ],
[0.16679579],
[0.43742415],
[0.11834401],
[0.79483396],
[0.19500774]],
[[0. ],
[0.07943237],
[0.5328288 ],
[0.77599657],
[0.63793445],
[0.19860882],
[0.11559258],
[0.9838874 ],
[0.25520208],
[0.5648425 ]],
[[0. ],
[0.05485739],
[0.68932915],
[0.46699798],
[0.42969576],
[0.18972297],
[0.46102318],
[0.9518804 ],
[0.49459723],
[0.08304291]],
[[0. ],
[0.3869838 ],
[0.80616343],
[0.29453617],
[0.90575564],
[0.9491408 ],
[0.0916879 ],
[0.36297712],
[0.9152429 ],
[0.67923504]],
[[0. ],
[0.22770868],
[0.04828448],
[0.59498733],
[0.8257741 ],
[0.77833605],
[0.30659854],
[0.30935422],
[0.96905756],
[0.3825807 ]],
[[0. ],
[0.2682056 ],
[0.12400189],
[0.14326625],
[0.13485809],
[0.60932964],
[0.49995294],
[0.48518673],
[0.9061716 ],
[0.6811738 ]],
[[0. ],
[0.5348281 ],
[0.4997965 ],
[0.63221145],
[0.46155205],
[0.7663521 ],
[0.91362387],
[0.2826063 ],
[0.68864125],
[0.6800111 ]],
[[0. ],
[0.39924976],
[0.45846376],
[0.7625721 ],
[0.99027014],
[0.39566088],
[0.94737506],
[0.4506151 ],
[0.24510984],
[0.89123327]]],
[[[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ],
[0. ]],
[[0. ],
[0.9952434 ],
[0.01445136],
[0.20864646],
[0.5734403 ],
[0.9085302 ],
[0.51775485],
[0.8855587 ],
[0.28934476],
[0.06262866]],
[[0. ],
[0.9793667 ],
[0.5434468 ],
[0.2679368 ],
[0.10213093],
[0.8542698 ],
[0.2450935 ],
[0.11804035],
[0.8925573 ],
[0.24042603]],
[[0. ],
[0.4573032 ],
[0.2561469 ],
[0.00524052],
[0.57438296],
[0.5396885 ],
[0.8146391 ],
[0.9708599 ],
[0.59276104],
[0.35178396]],
[[0. ],
[0.8042833 ],
[0.7458325 ],
[0.5945834 ],
[0.9203529 ],
[0.76014346],
[0.69867104],
[0.25335407],
[0.96289504],
[0.80579585]],
[[0. ],
[0.16236259],
[0.4947492 ],
[0.41423783],
[0.34527972],
[0.4402823 ],
[0.47581652],
[0.82424086],
[0.6161921 ],
[0.6575118 ]],
[[0. ],
[0.98747057],
[0.66196424],
[0.043115 ],
[0.9993771 ],
[0.12203664],
[0.7465267 ],
[0.3408288 ],
[0.94268566],
[0.8820024 ]],
[[0. ],
[0.9850347 ],
[0.5541368 ],
[0.07462564],
[0.07905959],
[0.1890236 ],
[0.6264842 ],
[0.36688676],
[0.6111452 ],
[0.20477323]],
[[0. ],
[0.07814959],
[0.49131715],
[0.14614591],
[0.7088246 ],
[0.77301365],
[0.7900142 ],
[0.6824042 ],
[0.02469799],
[0.20656963]],
[[0. ],
[0.27418277],
[0.2796882 ],
[0.7702565 ],
[0.9263017 ],
[0.96371156],
[0.9759365 ],
[0.46440127],
[0.12412024],
[0.4896511 ]]]], dtype=float32)>
Im searching in the MongoDB by using a GEOJSON polygon without holes. I get an error when accesing the cursor. The error is:
OperationFailure: Secondary loops not contained by first exterior loop - secondary loops must be holes.
Code:
import pymongo
import json
def loadjson(filename):
with open(filename) as f:
mydict = json.load(f)
return mydict
client = pymongo.MongoClient('mongodb://' \
+ 'username' \
+ ':' \
+ "password" + \
'#localhost:num')
gjOr=loadjson('polygon.json')
database = client.database
collection = database["name"]
filParam = {"field": {"$geoWithin": {"$geometry": gjOr["geometry"]}}}
exParam = {"parameter": 1}
cursorItineraries = collection.find(filParam, exParam).batch_size(2)
for i in cursorItineraries:
print(i)
Error :
OperationFailure: Secondary loops not contained by first exterior loop - secondary loops must be holes: [ [ 4.423298831017474, 45.92246510136637 ], [ 4.423272328729544, 45.9224124914541 ], [ 4.423296014293201, 45.92245885344253 ], [ 4.423298831017474, 45.92246510136637 ] ] first loop: [ [ 45.80122758057101, 4.376352681950241 ], [ 45.8044838524037, 4.376220111724892 ], [ 45.80502003601278, 4.384299007716068 ], [ 45.80303745929081, 4.388437890276315 ], [ 45.81747689026152, 4.380885175476993 ], [ 45.82029916312589, 4.38306756294014 ], [ 45.82485210019716, 4.37663747923372 ], [ 45.82534211272147, 4.381172591846549 ], [ 45.83402693020424, 4.384295932265717 ], [ 45.83750180635732, 4.391204268525621 ], [ 45.83984258047744, 4.387611887543129 ], [ 45.84600085935993, 4.383682442715087 ], [ 45.85031045846961, 4.391211993814656 ], [ 45.85985548506667, 4.396744466054351 ], [ 45.8651960481546, 4.395075656820409 ], [ 45.86464844812487, 4.390221929905438 ], [ 45.86619589736199, 4.361702458243602 ], [ 45.87693527428829, 4.343076054047009 ], [ 45.88006074780913, 4.340914771281566 ], [ 45.8901897554254, 4.337545959836689 ], [ 45.89739332872952, 4.326829637196063 ], [ 45.90391638987086, 4.32382549599472 ], [ 45.905667965693, 4.327990843251258 ], [ 45.91579311238129, 4.346817863136088 ], [ 45.91668891008069, 4.342257953265195 ], [ 45.9216251460626, 4.338982596932619 ], [ 45.92997139286699, 4.346628121156797 ], [ 45.93185900371628, 4.337155914633498 ], [ 45.94115045303277, 4.322661854063225 ], [ 45.9423274157955, 4.313001397538913 ], [ 45.95343445337902, 4.294525894838753 ], [ 45.95835570732176, 4.301188120177739 ], [ 45.9732026997347, 4.289431726797376 ], [ 45.97782302017173, 4.276356808225782 ], [ 45.97999822296752, 4.273060858987787 ], [ 45.98120074142516, 4.259590688670001 ], [ 45.98675497867428, 4.248658637365085 ], [ 45.99308572400004, 4.247947359178051 ], [ 45.99614759409149, 4.246662259181909 ], [ 45.99563177014321, 4.251765125404147 ], [ 45.99854856923263, 4.265285854935993 ], [ 45.99274541001228, 4.270233183990548 ], [ 45.99239729542708, 4.275045816032846 ], [ 45.99568744605374, 4.289609476470726 ], [ 45.98892048061472, 4.300698529315564 ], [ 45.99439301256719, 4.304534538221632 ], [ 46.0051569133273, 4.31261301303475 ], [ 46.00804632488756, 4.310942716001039 ], [ 46.01128401825687, 4.308859856852684 ], [ 46.02162453052691, 4.306458712603317 ], [ 46.0215031603917, 4.305677962950914 ], [ 46.02080232740203, 4.301223173083229 ], [ 46.01839338755868, 4.288344635543249 ], [ 46.0218025518577, 4.28062738017141 ], [ 46.02393104466132, 4.276598681060639 ], [ 46.03355588764234, 4.270436612377799 ], [ 46.03608569278646, 4.261172108339184 ], [ 46.03914954925174, 4.261574439856753 ], [ 46.04921372290215, 4.251694718008523 ], [ 46.05182172309517, 4.254034942879366 ], [ 46.05394712145236, 4.25770790411945 ], [ 46.05200674707666, 4.266837389513737 ], [ 46.05371908926419, 4.285713698620556 ], [ 46.06348629342862, 4.297946350027559 ], [ 46.06607892681861, 4.300652169094047 ], [ 46.07849083960166, 4.302039193299913 ], [ 46.07922531473847, 4.305599848572079 ], [ 46.08175515713837, 4.311015193331216 ], [ 46.08505691781212, 4.310620920782156 ], [ 46.09488700704484, 4.311938135622067 ], [ 46.10036561585792, 4.306608118283756 ], [ 46.11003344270973, 4.30990269743367 ], [ 46.11227779249233, 4.313417891784874 ], [ 46.11685288921163, 4.320328061893218 ], [ 46.13001463483668, 4.322725351722958 ], [ 46.13326362593953, 4.330293102924566 ], [ 46.13554160597837, 4.333348440257132 ], [ 46.13830957901551, 4.357233936672898 ], [ 46.13849351079732, 4.362195204890961 ], [ 46.1498366233474, 4.377795652104825 ], [ 46.14969360157965, 4.382423398963222 ], [ 46.14767758414561, 4.386051143732711 ], [ 46.13709181954217, 4.408851926159123 ], [ 46.13594976925808, 4.417961239075054 ], [ 46.15142016044436, 4.424267792344496 ], [ 46.15560425283834, 4.431770589521594 ], [ 46.16791483778646, 4.439193639737003 ], [ 46.16998904853596, 4.435772060175736 ], [ 46.18010788550735, 4.414915433192237 ], [ 46.18156481929947, 4.411150008217347 ], [ 46.18227866102438, 4.406966209179808 ], [ 46.18456671088968, 4.403509174410789 ], [ 46.18925424643544, 4.396783755671023 ], [ 46.19570191957066, 4.39747939068123 ], [ 46.20316836045577, 4.423571836902974 ], [ 46.20706171749928, 4.422359640985007 ], [ 46.21010077361071, 4.420471659265854 ], [ 46.21965844746768, 4.407356367902624 ], [ 46.21982737938801, 4.388683312501862 ], [ 46.22571105646608, 4.386829907900418 ], [ 46.23755075963317, 4.39209447339758 ], [ 46.24354546160409, 4.388785758384324 ], [ 46.24619830411654, 4.386173183144986 ], [ 46.24794900440396, 4.38889827660288 ], [ 46.2643415894465, 4.393582057210332 ], [ 46.26655827183746, 4.397449686073675 ], [ 46.27187809126278, 4.397064201352286 ], [ 46.27457798360896, 4.396974174839016 ], [ 46.28435539290889, 4.399371429518632 ], [ 46.29277972056357, 4.407707294944299 ], [ 46.29609636649197, 4.406422030172297 ], [ 46.29537513021594, 4.411216732267508 ], [ 46.29633635905063, 4.424994132856644 ], [ 46.30286120593534, 4.427650449880751 ], [ 46.29631496657498, 4.438330372797787 ], [ 46.29309133554136, 4.440062626728649 ], [ 46.29647617285935, 4.453855067913997 ], [ 46.29405157990985, 4.46279605745037 ], [ 46.28443417433566, 4.47602647504644 ], [ 46.28709961733546, 4.484382424854218 ], [ 46.28803138168911, 4.489069361733742 ], [ 46.27010728719618, 4.50209226866315 ], [ 46.26716933623024, 4.504596974106646 ], [ 46.2719961499778, 4.542944499553459 ], [ 46.27395480446275, 4.547049893978835 ], [ 46.2832619806662, 4.548685286189365 ], [ 46.29129615541792, 4.556044471210963 ], [ 46.29402942721709, 4.55844617134213 ], [ 46.29306430926767, 4.570871194910193 ], [ 46.29001185910924, 4.572758405594865 ], [ 46.27707566137231, 4.573323169325609 ], [ 46.27025349602457, 4.58326888257721 ], [ 46.26839674289919, 4.587119484978682 ], [ 46.26643179980633, 4.615050419767011 ], [ 46.2654086056003, 4.617672483965549 ], [ 45.80592709186426, 4.865072663474678 ], [ 45.80434184846744, 4.864499321786514 ], [ 45.80433950695036, 4.864498347340356 ], [ 45.80545594864911, 4.848595888972264 ], [ 45.76435319429793, 4.801430760920382 ], [ 45.71917418711324, 4.781774751613478 ], [ 45.68363470756887, 4.80870542554455 ], [ 45.80122758057101, 4.376352681950241 ] ], full error: {'ok': 0.0, 'errmsg': 'Secondary loops not contained by first exterior loop - secondary loops must be holes: [ [ 4.423298831017474, 45.92246510136637 ], [ 4.423272328729544, 45.9224124914541 ], [ 4.423296014293201, 45.92245885344253 ], [ 4.423298831017474, 45.92246510136637 ] ] first loop: [ [ 45.80122758057101, 4.376352681950241 ], [ 45.8044838524037, 4.376220111724892 ], [ 45.80502003601278, 4.384299007716068 ], [ 45.80303745929081, 4.388437890276315 ], [ 45.81747689026152, 4.380885175476993 ], [ 45.82029916312589, 4.38306756294014 ], [ 45.82485210019716, 4.37663747923372 ], [ 45.82534211272147, 4.381172591846549 ], [ 45.83402693020424, 4.384295932265717 ], [ 45.83750180635732, 4.391204268525621 ], [ 45.83984258047744, 4.387611887543129 ], [ 45.84600085935993, 4.383682442715087 ], [ 45.85031045846961, 4.391211993814656 ], [ 45.85985548506667, 4.396744466054351 ], [ 45.8651960481546, 4.395075656820409 ], [ 45.86464844812487, 4.390221929905438 ], [ 45.86619589736199, 4.361702458243602 ], [ 45.87693527428829, 4.343076054047009 ], [ 45.88006074780913, 4.340914771281566 ], [ 45.8901897554254, 4.337545959836689 ], [ 45.89739332872952, 4.326829637196063 ], [ 45.90391638987086, 4.32382549599472 ], [ 45.905667965693, 4.327990843251258 ], [ 45.91579311238129, 4.346817863136088 ], [ 45.91668891008069, 4.342257953265195 ], [ 45.9216251460626, 4.338982596932619 ], [ 45.92997139286699, 4.346628121156797 ], [ 45.93185900371628, 4.337155914633498 ], [ 45.94115045303277, 4.322661854063225 ], [ 45.9423274157955, 4.313001397538913 ], [ 45.95343445337902, 4.294525894838753 ], [ 45.95835570732176, 4.301188120177739 ], [ 45.9732026997347, 4.289431726797376 ], [ 45.97782302017173, 4.276356808225782 ], [ 45.97999822296752, 4.273060858987787 ], [ 45.98120074142516, 4.259590688670001 ], [ 45.98675497867428, 4.248658637365085 ], [ 45.99308572400004, 4.247947359178051 ], [ 45.99614759409149, 4.246662259181909 ], [ 45.99563177014321, 4.251765125404147 ], [ 45.99854856923263, 4.265285854935993 ], [ 45.99274541001228, 4.270233183990548 ], [ 45.99239729542708, 4.275045816032846 ], [ 45.99568744605374, 4.289609476470726 ], [ 45.98892048061472, 4.300698529315564 ], [ 45.99439301256719, 4.304534538221632 ], [ 46.0051569133273, 4.31261301303475 ], [ 46.00804632488756, 4.310942716001039 ], [ 46.01128401825687, 4.308859856852684 ], [ 46.02162453052691, 4.306458712603317 ], [ 46.0215031603917, 4.305677962950914 ], [ 46.02080232740203, 4.301223173083229 ], [ 46.01839338755868, 4.288344635543249 ], [ 46.0218025518577, 4.28062738017141 ], [ 46.02393104466132, 4.276598681060639 ], [ 46.03355588764234, 4.270436612377799 ], [ 46.03608569278646, 4.261172108339184 ], [ 46.03914954925174, 4.261574439856753 ], [ 46.04921372290215, 4.251694718008523 ], [ 46.05182172309517, 4.254034942879366 ], [ 46.05394712145236, 4.25770790411945 ], [ 46.05200674707666, 4.266837389513737 ], [ 46.05371908926419, 4.285713698620556 ], [ 46.06348629342862, 4.297946350027559 ], [ 46.06607892681861, 4.300652169094047 ], [ 46.07849083960166, 4.302039193299913 ], [ 46.07922531473847, 4.305599848572079 ], [ 46.08175515713837, 4.311015193331216 ], [ 46.08505691781212, 4.310620920782156 ], [ 46.09488700704484, 4.311938135622067 ], [ 46.10036561585792, 4.306608118283756 ], [ 46.11003344270973, 4.30990269743367 ], [ 46.11227779249233, 4.313417891784874 ], [ 46.11685288921163, 4.320328061893218 ], [ 46.13001463483668, 4.322725351722958 ], [ 46.13326362593953, 4.330293102924566 ], [ 46.13554160597837, 4.333348440257132 ], [ 46.13830957901551, 4.357233936672898 ], [ 46.13849351079732, 4.362195204890961 ], [ 46.1498366233474, 4.377795652104825 ], [ 46.14969360157965, 4.382423398963222 ], [ 46.14767758414561, 4.386051143732711 ], [ 46.13709181954217, 4.408851926159123 ], [ 46.13594976925808, 4.417961239075054 ], [ 46.15142016044436, 4.424267792344496 ], [ 46.15560425283834, 4.431770589521594 ], [ 46.16791483778646, 4.439193639737003 ], [ 46.16998904853596, 4.435772060175736 ], [ 46.18010788550735, 4.414915433192237 ], [ 46.18156481929947, 4.411150008217347 ], [ 46.18227866102438, 4.406966209179808 ], [ 46.18456671088968, 4.403509174410789 ], [ 46.18925424643544, 4.396783755671023 ], [ 46.19570191957066, 4.39747939068123 ], [ 46.20316836045577, 4.423571836902974 ], [ 46.20706171749928, 4.422359640985007 ], [ 46.21010077361071, 4.420471659265854 ], [ 46.21965844746768, 4.407356367902624 ], [ 46.21982737938801, 4.388683312501862 ], [ 46.22571105646608, 4.386829907900418 ], [ 46.23755075963317, 4.39209447339758 ], [ 46.24354546160409, 4.388785758384324 ], [ 46.24619830411654, 4.386173183144986 ], [ 46.24794900440396, 4.38889827660288 ], [ 46.2643415894465, 4.393582057210332 ], [ 46.26655827183746, 4.397449686073675 ], [ 46.27187809126278, 4.397064201352286 ], [ 46.27457798360896, 4.396974174839016 ], [ 46.28435539290889, 4.399371429518632 ], [ 46.29277972056357, 4.407707294944299 ], [ 46.29609636649197, 4.406422030172297 ], [ 46.29537513021594, 4.411216732267508 ], [ 46.29633635905063, 4.424994132856644 ], [ 46.30286120593534, 4.427650449880751 ], [ 46.29631496657498, 4.438330372797787 ], [ 46.29309133554136, 4.440062626728649 ], [ 46.29647617285935, 4.453855067913997 ], [ 46.29405157990985, 4.46279605745037 ], [ 46.28443417433566, 4.47602647504644 ], [ 46.28709961733546, 4.484382424854218 ], [ 46.28803138168911, 4.489069361733742 ], [ 46.27010728719618, 4.50209226866315 ], [ 46.26716933623024, 4.504596974106646 ], [ 46.2719961499778, 4.542944499553459 ], [ 46.27395480446275, 4.547049893978835 ], [ 46.2832619806662, 4.548685286189365 ], [ 46.29129615541792, 4.556044471210963 ], [ 46.29402942721709, 4.55844617134213 ], [ 46.29306430926767, 4.570871194910193 ], [ 46.29001185910924, 4.572758405594865 ], [ 46.27707566137231, 4.573323169325609 ], [ 46.27025349602457, 4.58326888257721 ], [ 46.26839674289919, 4.587119484978682 ], [ 46.26643179980633, 4.615050419767011 ], [ 46.2654086056003, 4.617672483965549 ], [ 45.80592709186426, 4.865072663474678 ], [ 45.80434184846744, 4.864499321786514 ], [ 45.80433950695036, 4.864498347340356 ], [ 45.80545594864911, 4.848595888972264 ], [ 45.76435319429793, 4.801430760920382 ], [ 45.71917418711324, 4.781774751613478 ], [ 45.68363470756887, 4.80870542554455 ], [ 45.80122758057101, 4.376352681950241 ] ]', 'code': 2, 'codeName': 'BadValue', 'operationTime': Timestamp(1611827959, 1), '$clusterTime': {'clusterTime': Timestamp(1611827959, 1), 'signature': {'hash': b'\x8d\xb6\x9b7\x0b\xe7!\x11\xe6\xc1\x13\x06\xf4\xe0\xca\xfe\xcbN\x0cy', 'keyId': 6872375198939611144}}}
In R, I can easily get the performance of a random forest like the following.
How can I get the similar stuff in Python easily? Thanks a lot.
Summary of the Random Forest Model
==================================
Number of observations used to build the model: 35
Missing value imputation is active.
Call:
randomForest(formula = rank ~ .,
data = crs$dataset[crs$sample, c(crs$input, crs$target)],
ntree = 500, mtry = 3, importance = TRUE, replace = FALSE, na.action = na.roughfix)
Type of random forest: regression
Number of trees: 500
No. of variables tried at each split: 3
Mean of squared residuals: 5.578147
% Var explained: 97.22
Variable Importance
Here there is a simple example using sklearn random forest . http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier
You can easily get the values that you are looking for.
In this example we are using an input matrix X with 2 variables and a binary output y.
from sklearn.ensemble import RandomForestClassifier
fr = RandomForestClassifier(n_estimators=100, oob_score=True).fit(X, y)
fr.n_estimators
Out[10]: 100
fr.oob_decision_function_
Out[11]:
array([[ 0.14285714, 0.85714286],
[ 0.86666667, 0.13333333],
[ 0.02631579, 0.97368421],
[ 1. , 0. ],
[ 0.97826087, 0.02173913],
[ 0.97826087, 0.02173913],
[ 0.20512821, 0.79487179],
[ 0.97368421, 0.02631579],
[ 0.77777778, 0.22222222],
[ 0. , 1. ],
[ 0. , 1. ],
[ 1. , 0. ],
[ 0.52380952, 0.47619048],
[ 0.43243243, 0.56756757],
[ 1. , 0. ],
[ 0. , 1. ],
[ 0.05714286, 0.94285714],
[ 0. , 1. ],
[ 1. , 0. ],
[ 0.76470588, 0.23529412],
[ 1. , 0. ],
[ 0. , 1. ],
[ 0.95454545, 0.04545455],
[ 0.9 , 0.1 ],
[ 0.02222222, 0.97777778],
[ 0.875 , 0.125 ],
[ 0.02857143, 0.97142857],
[ 1. , 0. ],
[ 0.58823529, 0.41176471],
[ 0. , 1. ],
[ 0.20512821, 0.79487179],
[ 0.97435897, 0.02564103],
[ 0.91176471, 0.08823529],
[ 0. , 1. ],
[ 0.30232558, 0.69767442],
[ 1. , 0. ],
[ 0.94444444, 0.05555556],
[ 0. , 1. ],
[ 0.075 , 0.925 ],
[ 0.05263158, 0.94736842],
[ 1. , 0. ],
[ 0. , 1. ],
[ 0.02702703, 0.97297297],
[ 0.91176471, 0.08823529],
[ 0.43243243, 0.56756757],
[ 0.08333333, 0.91666667],
[ 0.10526316, 0.89473684],
[ 0.93548387, 0.06451613],
[ 0.02857143, 0.97142857],
[ 0.53658537, 0.46341463],
[ 0.5 , 0.5 ],
[ 0.66666667, 0.33333333],
[ 1. , 0. ],
[ 0.55555556, 0.44444444],
[ 0.96666667, 0.03333333],
[ 0.97142857, 0.02857143],
[ 0. , 1. ],
[ 0. , 1. ],
[ 1. , 0. ],
[ 0.05882353, 0.94117647],
[ 0.94594595, 0.05405405],
[ 0.11904762, 0.88095238],
[ 0.92307692, 0.07692308],
[ 0.69767442, 0.30232558],
[ 1. , 0. ],
[ 0.12121212, 0.87878788],
[ 1. , 0. ],
[ 0.97727273, 0.02272727],
[ 1. , 0. ],
[ 0.87878788, 0.12121212],
[ 0.02380952, 0.97619048],
[ 0. , 1. ],
[ 0. , 1. ],
[ 0. , 1. ],
[ 0.10810811, 0.89189189],
[ 1. , 0. ],
[ 1. , 0. ],
[ 0.97619048, 0.02380952],
[ 0.54545455, 0.45454545],
[ 0.02380952, 0.97619048],
[ 0.07317073, 0.92682927],
[ 0.94285714, 0.05714286],
[ 0.25714286, 0.74285714],
[ 0. , 1. ],
[ 0. , 1. ],
[ 0.97560976, 0.02439024],
[ 0.11111111, 0.88888889],
[ 1. , 0. ],
[ 1. , 0. ],
[ 0.02857143, 0.97142857],
[ 0.97916667, 0.02083333],
[ 0. , 1. ],
[ 0.02564103, 0.97435897],
[ 0. , 1. ],
[ 0.32258065, 0.67741935],
[ 0.56410256, 0.43589744],
[ 1. , 0. ],
[ 0.92682927, 0.07317073],
[ 1. , 0. ],
[ 0.08823529, 0.91176471]])
fr.oob_score_
Out[12]: 0.87
fr.feature_importances_
Out[13]: array([ 0.82407373, 0.17592627])
Say I need to max_(a', m') f(a, m, e, m', a'), and I have approximated f with a grid V1. This is a numpy matrix with shape (nA, nM, nE, nM, nA) (attached in the end).
I want to first interpolate and then do the maximization. The following is my current code (I paste code to recreate Grid in the end):
# takes grid indices (first three dimensions) idx and interpolates on V
def interpolateV(idx, V, Grid):
from scipy.interpolate import interp2d
f = interp2d(Grid.mGrid, Grid.aGrid, V[idx])
return f
# (somewhere else:)
s2 = (Grid.nM, Grid.nA, Grid.nE)
v1Max = np.empty(s2)
v1ArgMaxA = np.empty(s2)
v1ArgMaxM = np.empty(s2)
from scipy import optimize
for idx in np.ndindex(V1[..., 0,0].shape):
V1i = interpolateV(idx, V1, Grid)
x, f, d = optimize.fmin_l_bfgs_b(lambda x: -V1i(x[0], x[1]), np.array([1, 1]), bounds=[(Grid.aMin, Grid.aMax), (Grid.mMin, Grid.mMax)], approx_grad=True)
v1Max[idx] = f
v1ArgMaxA[idx], v1ArgMaxM[idx] = x
# let's compare with standard grid-wise optimization (without interpolation):
temp = V1.max(axis=-1)
# maximize over m
v1Max = temp.max(axis=-1)
# now max over a, given optimal m
v1ArgMaxAGrid = temp.argmax(axis=-1)
So far, so good. However, the values of the interpolated maximized are way off:
In[51]: v1ArgMaxAGrid[:,:,0]
Out[51]:
array([[0, 0, 0, 0, 2],
[0, 0, 0, 0, 2],
[0, 0, 0, 2, 2],
[0, 0, 0, 2, 3],
[0, 0, 0, 2, 3]], dtype=int64)
In[54]: Grid.aGrid[v1ArgMaxAGrid[:,:,0]]
Out[54]:
array([[ 0. , 0. , 0. , 0. , 3.5 ],
[ 0. , 0. , 0. , 0. , 3.5 ],
[ 0. , 0. , 0. , 3.5 , 3.5 ],
[ 0. , 0. , 0. , 3.5 , 5.25],
[ 0. , 0. , 0. , 3.5 , 5.25]])
In[52]: v1ArgMaxA[:,:,0]
Out[52]:
array([[ 0. , 0.75 , 2.25 , 7. , 7. ],
[ 0. , 1.5 , 4.247, 7. , 7. ],
[ 0.75 , 2.25 , 7. , 7. , 7. ],
[ 1.5 , 1.5 , 7. , 7. , 7. ],
[ 2.25 , 4.939, 7. , 7. , 7. ]])
What is happening here; why are the values so off? Am I doing a mistake?
following the copy paste to recreate Grid, V1:
class Grids(object):
nE = 2
nA = 5
nM = 5
M = 3
A = 7
mMin = 0
mMax = M
aMin = 0
aMax = A
def __init__(self):
self.reset();
def reset(self):
self.mGrid = np.linspace(self.mMin, self.mMax, self.nM)
self.aGrid = np.linspace(self.aMin, self.aMax, self.nA)
self.eGrid = np.array([0.318, 3.149])
self.transitionE = np.array([[1., 0.],
[0., 1.]])
import numpy as np
Grid = Grids()
V1 = np.array([[[[[ 1.19 , 0.975, -0.371, -2.848, -6.456],
[ -1.463, -4.313, -8.294, -13.407, -19.65 ],
[ -9.888, -15.377, -21.997, -29.748, -38.63 ],
[-24.574, -32.701, -41.958, -52.347, -63.866],
[-45.562, -56.325, -68.218, -81.242, -95.397]],
[[ 64.724, 64.672, 64.567, 64.358, 54.127],
[ 64.247, 63.964, 53.759, 52.687, 50.487],
[ 53.526, 52.078, 49.501, 45.799, 40.969],
[ 48.389, 44.307, 39.105, 32.769, 25.314],
[ 37.062, 30.347, 22.52 , 13.553, 3.47 ]]],
[[[ 12.624, 12.704, 12.591, 2.602, 1.618],
[ 2.237, 2.011, 0.655, -1.832, -5.45 ],
[ -0.064, -2.928, -6.923, -12.049, -18.306],
[ -8.624, -14.126, -20.759, -28.522, -37.416],
[-23.488, -31.625, -40.894, -51.293, -62.822]],
[[ 65.686, 65.695, 65.679, 65.631, 65.537],
[ 65.401, 65.342, 65.23 , 65.014, 54.778],
[ 65.174, 64.881, 54.667, 53.59 , 51.385],
[ 54.43 , 52.973, 50.396, 46.685, 41.855],
[ 49.228, 45.138, 39.936, 33.594, 26.136]]],
[[[ 13.681, 13.872, 14.024, 14.117, 14.093],
[ 13.671, 13.74 , 13.617, 3.617, 2.624],
[ 3.636, 3.397, 2.027, -0.474, -4.106],
[ 1.2 , -1.677, -5.684, -10.823, -17.092],
[ -7.538, -13.051, -19.694, -27.468, -36.373]],
[[ 66.553, 66.597, 66.623, 66.631, 66.614],
[ 66.362, 66.364, 66.342, 66.287, 66.188],
[ 66.327, 66.259, 66.138, 65.917, 55.676],
[ 66.077, 65.776, 55.562, 54.476, 52.271],
[ 55.269, 53.804, 51.227, 47.51 , 42.677]]],
[[[ 14.6 , 14.839, 15.054, 15.242, 15.394],
[ 14.728, 14.909, 15.05 , 15.133, 15.098],
[ 15.07 , 15.126, 14.989, 4.975, 3.968],
[ 4.9 , 4.648, 3.265, 0.752, -2.892],
[ 2.286, -0.601, -4.62 , -9.769, -16.048]],
[[ 67.36 , 67.427, 67.481, 67.52 , 67.543],
[ 67.229, 67.266, 67.286, 67.287, 67.265],
[ 67.288, 67.281, 67.25 , 67.19 , 67.085],
[ 67.231, 67.154, 67.033, 66.803, 56.562],
[ 66.917, 66.607, 56.393, 55.301, 53.093]]],
[[[ 15.442, 15.71 , 15.96 , 16.191, 16.4 ],
[ 15.647, 15.875, 16.08 , 16.257, 16.399],
[ 16.128, 16.294, 16.422, 16.491, 16.443],
[ 16.334, 16.377, 16.227, 6.201, 5.182],
[ 5.986, 5.723, 4.33 , 1.806, -1.849]],
[[ 68.123, 68.207, 68.28 , 68.342, 68.391],
[ 68.036, 68.096, 68.143, 68.176, 68.194],
[ 68.155, 68.183, 68.195, 68.19 , 68.163],
[ 68.192, 68.176, 68.145, 68.076, 67.971],
[ 68.07 , 67.984, 67.864, 67.628, 57.384]]]],
[[[[ 11.877, 1.81 , 1.59 , 0.238, -2.246],
[ 0.873, -0.853, -3.709, -7.696, -12.814],
[ -4.928, -9.292, -14.787, -21.413, -29.17 ],
[-16.988, -23.99 , -32.123, -41.386, -51.78 ],
[-35.352, -44.989, -55.758, -67.657, -80.686]],
[[ 65.151, 65.131, 65.075, 64.966, 64.753],
[ 64.779, 64.647, 64.36 , 54.151, 53.076],
[ 54.24 , 53.917, 52.465, 49.888, 46.183],
[ 51.728, 48.771, 44.694, 39.483, 33.153],
[ 43.026, 37.436, 30.734, 22.892, 13.934]]],
[[[ 13.101, 13.245, 13.318, 13.2 , 3.204],
[ 12.924, 2.847, 2.616, 1.253, -1.24 ],
[ 2.272, 0.533, -2.337, -6.338, -11.47 ],
[ -3.664, -8.041, -13.548, -20.187, -27.956],
[-15.902, -22.915, -31.058, -40.332, -50.737]],
[[ 66.066, 66.092, 66.098, 66.078, 66.025],
[ 65.827, 65.8 , 65.738, 65.622, 65.403],
[ 65.706, 65.564, 65.268, 55.054, 53.974],
[ 55.144, 54.812, 53.36 , 50.774, 47.069],
[ 52.567, 49.602, 45.525, 40.308, 33.975]]],
[[[ 14.087, 14.302, 14.487, 14.632, 14.72 ],
[ 14.148, 14.281, 14.344, 14.216, 4.21 ],
[ 14.323, 4.232, 3.987, 2.611, 0.104],
[ 3.536, 1.784, -1.099, -5.112, -10.256],
[ -2.578, -6.965, -12.484, -19.133, -26.912]],
[[ 66.905, 66.96 , 66.999, 67.022, 67.025],
[ 66.742, 66.762, 66.76 , 66.734, 66.676],
[ 66.754, 66.718, 66.646, 66.525, 66.301],
[ 66.609, 66.459, 66.163, 55.94 , 54.86 ],
[ 55.983, 55.643, 54.191, 51.599, 47.891]]],
[[[ 14.969, 15.221, 15.454, 15.663, 15.844],
[ 15.134, 15.338, 15.513, 15.648, 15.725],
[ 15.548, 15.667, 15.716, 15.574, 5.554],
[ 15.587, 5.483, 5.226, 3.837, 1.318],
[ 4.622, 2.859, -0.034, -4.058, -9.213]],
[[ 67.691, 67.766, 67.829, 67.879, 67.915],
[ 67.581, 67.629, 67.662, 67.678, 67.676],
[ 67.669, 67.679, 67.669, 67.637, 67.574],
[ 67.658, 67.612, 67.541, 67.411, 67.187],
[ 67.449, 67.29 , 66.994, 56.765, 55.682]]],
[[[ 15.786, 16.063, 16.324, 16.569, 16.794],
[ 16.016, 16.258, 16.48 , 16.679, 16.85 ],
[ 16.533, 16.724, 16.884, 17.006, 17.07 ],
[ 16.811, 16.918, 16.954, 16.8 , 6.768],
[ 16.673, 6.559, 6.29 , 4.891, 2.362]],
[[ 68.439, 68.529, 68.61 , 68.679, 68.737],
[ 68.368, 68.436, 68.492, 68.535, 68.566],
[ 68.507, 68.546, 68.571, 68.581, 68.574],
[ 68.572, 68.573, 68.563, 68.523, 68.46 ],
[ 68.497, 68.443, 68.372, 68.236, 68.009]]]],
[[[[ 12.453, 12.498, 2.425, 2.198, 0.84 ],
[ 2.083, 1.483, -0.248, -3.111, -7.104],
[ -1.092, -4.331, -8.701, -14.202, -20.834],
[-10.528, -16.405, -23.412, -31.551, -40.82 ],
[-26.266, -34.779, -44.422, -55.196, -67.101]],
[[ 65.555, 65.558, 65.534, 65.474, 65.36 ],
[ 65.252, 65.179, 65.043, 64.752, 54.54 ],
[ 64.974, 54.631, 54.304, 52.852, 50.272],
[ 53.942, 52.11 , 49.158, 45.072, 39.867],
[ 47.865, 43.4 , 37.823, 31.106, 23.273]]],
[[[ 13.541, 13.722, 13.859, 13.927, 13.802],
[ 13.5 , 13.534, 3.451, 3.214, 1.846],
[ 3.482, 2.868, 1.123, -1.753, -5.76 ],
[ 0.172, -3.08 , -7.463, -12.976, -19.62 ],
[ -9.442, -15.329, -22.348, -30.497, -39.776]],
[[ 66.433, 66.473, 66.495, 66.496, 66.472],
[ 66.231, 66.227, 66.196, 66.13 , 66.011],
[ 66.178, 66.096, 65.952, 65.655, 55.438],
[ 65.877, 55.526, 55.199, 53.738, 51.158],
[ 54.781, 52.941, 49.989, 45.897, 40.689]]],
[[[ 14.475, 14.708, 14.917, 15.095, 15.235],
[ 14.588, 14.759, 14.886, 14.943, 14.808],
[ 14.899, 14.92 , 4.823, 4.572, 3.19 ],
[ 4.746, 4.119, 2.362, -0.527, -4.546],
[ 1.258, -2.005, -6.398, -11.922, -18.577]],
[[ 67.247, 67.312, 67.362, 67.398, 67.417],
[ 67.109, 67.142, 67.158, 67.152, 67.123],
[ 67.158, 67.145, 67.105, 67.033, 66.909],
[ 67.082, 66.991, 66.847, 66.541, 56.324],
[ 66.717, 56.357, 56.03 , 54.563, 51.98 ]]],
[[[ 15.325, 15.59 , 15.836, 16.062, 16.265],
[ 15.522, 15.744, 15.943, 16.111, 16.24 ],
[ 15.987, 16.144, 16.257, 16.301, 16.152],
[ 16.163, 16.171, 6.061, 5.798, 4.404],
[ 5.832, 5.195, 3.426, 0.527, -3.502]],
[[ 68.016, 68.098, 68.169, 68.228, 68.274],
[ 67.924, 67.981, 68.025, 68.054, 68.067],
[ 68.036, 68.059, 68.066, 68.056, 68.021],
[ 68.061, 68.039, 68. , 67.919, 67.794],
[ 67.921, 67.822, 67.677, 67.366, 57.146]]],
[[[ 16.121, 16.406, 16.678, 16.933, 17.171],
[ 16.372, 16.626, 16.862, 17.078, 17.271],
[ 16.921, 17.13 , 17.314, 17.469, 17.585],
[ 17.251, 17.395, 17.496, 17.527, 17.366],
[ 17.249, 17.246, 7.126, 6.852, 5.447]],
[[ 68.749, 68.846, 68.932, 69.008, 69.074],
[ 68.692, 68.768, 68.832, 68.884, 68.925],
[ 68.85 , 68.898, 68.934, 68.957, 68.965],
[ 68.939, 68.954, 68.961, 68.941, 68.906],
[ 68.901, 68.87 , 68.831, 68.744, 68.617]]]],
[[[[ 12.947, 13.074, 13.112, 3.034, 2.8 ],
[ 12.693, 2.693, 2.087, 0.35 , -2.518],
[ 1.618, -0.496, -3.741, -8.117, -13.624],
[ -5.192, -9.944, -15.827, -22.84 , -30.984],
[-18.306, -25.693, -34.212, -43.861, -54.64 ]],
[[ 65.941, 65.962, 65.961, 65.932, 65.868],
[ 65.688, 65.652, 65.575, 65.435, 65.141],
[ 65.537, 65.364, 55.018, 54.691, 53.236],
[ 55.031, 54.324, 52.497, 49.536, 45.456],
[ 51.579, 48.239, 43.787, 38.195, 31.487]]],
[[[ 13.954, 14.162, 14.337, 14.468, 14.529],
[ 13.994, 14.11 , 14.139, 4.049, 3.806],
[ 14.093, 4.079, 3.459, 1.708, -1.174],
[ 2.882, 0.755, -2.502, -6.891, -12.41 ],
[ -4.106, -8.869, -14.762, -21.786, -29.941]],
[[ 66.789, 66.84 , 66.876, 66.893, 66.891],
[ 66.617, 66.631, 66.623, 66.588, 66.519],
[ 66.614, 66.569, 66.484, 66.339, 66.039],
[ 66.441, 66.259, 55.913, 55.577, 54.122],
[ 55.87 , 55.155, 53.328, 50.361, 46.278]]],
[[[ 14.847, 15.096, 15.323, 15.525, 15.698],
[ 15.001, 15.198, 15.363, 15.484, 15.535],
[ 15.394, 15.496, 15.51 , 5.407, 5.15 ],
[ 15.356, 5.33 , 4.697, 2.934, 0.04 ],
[ 3.968, 1.831, -1.438, -5.837, -11.366]],
[[ 67.582, 67.654, 67.714, 67.761, 67.792],
[ 67.465, 67.509, 67.538, 67.549, 67.542],
[ 67.543, 67.548, 67.532, 67.492, 67.417],
[ 67.518, 67.463, 67.379, 67.224, 66.925],
[ 67.28 , 67.09 , 56.744, 56.402, 54.944]]],
[[[ 15.672, 15.946, 16.204, 16.444, 16.665],
[ 15.894, 16.132, 16.349, 16.541, 16.703],
[ 16.4 , 16.584, 16.734, 16.842, 16.879],
[ 16.657, 16.746, 16.749, 6.633, 6.364],
[ 16.443, 6.405, 5.762, 3.988, 1.083]],
[[ 68.334, 68.423, 68.501, 68.568, 68.623],
[ 68.258, 68.324, 68.377, 68.417, 68.443],
[ 68.391, 68.426, 68.447, 68.453, 68.439],
[ 68.447, 68.443, 68.427, 68.378, 68.302],
[ 68.357, 68.294, 68.209, 68.049, 67.747]]],
[[[ 16.448, 16.742, 17.021, 17.286, 17.535],
[ 16.719, 16.983, 17.23 , 17.46 , 17.67 ],
[ 17.294, 17.517, 17.72 , 17.899, 18.048],
[ 17.664, 17.835, 17.973, 18.068, 18.093],
[ 17.744, 17.822, 17.813, 7.687, 7.408]],
[[ 69.055, 69.156, 69.248, 69.331, 69.403],
[ 69.01 , 69.092, 69.163, 69.224, 69.273],
[ 69.184, 69.241, 69.286, 69.32 , 69.341],
[ 69.295, 69.321, 69.341, 69.339, 69.325],
[ 69.286, 69.274, 69.257, 69.203, 69.125]]]],
[[[[ 13.398, 13.568, 13.688, 13.721, 3.636],
[ 13.32 , 13.303, 3.298, 2.685, 0.942],
[ 3.204, 2.215, 0.095, -3.156, -7.538],
[ -0.982, -4.609, -9.366, -15.255, -22.274],
[-11.47 , -17.733, -25.126, -33.65 , -43.305]],
[[ 66.312, 66.348, 66.364, 66.359, 66.327],
[ 66.099, 66.088, 66.048, 65.967, 65.825],
[ 66.025, 65.928, 65.752, 55.405, 55.075],
[ 65.656, 55.413, 54.711, 52.875, 49.92 ],
[ 54.168, 51.953, 48.626, 44.159, 38.576]]],
[[[ 14.347, 14.575, 14.776, 14.945, 15.07 ],
[ 14.445, 14.605, 14.714, 14.737, 4.642],
[ 14.72 , 14.689, 4.669, 4.043, 2.286],
[ 4.468, 3.466, 1.333, -1.93 , -6.324],
[ 0.104, -3.533, -8.302, -14.201, -21.23 ]],
[[ 67.134, 67.195, 67.243, 67.274, 67.288],
[ 66.988, 67.017, 67.027, 67.015, 66.978],
[ 67.025, 67.005, 66.956, 66.871, 66.722],
[ 66.929, 66.822, 66.646, 56.291, 55.961],
[ 66.496, 56.244, 55.542, 53.7 , 50.742]]],
[[[ 15.208, 15.468, 15.71 , 15.931, 16.128],
[ 15.394, 15.611, 15.802, 15.961, 16.076],
[ 15.844, 15.99 , 16.086, 16.095, 5.986],
[ 15.983, 15.94 , 5.908, 5.269, 3.5 ],
[ 5.554, 4.541, 2.398, -0.876, -5.281]],
[[ 67.909, 67.988, 68.057, 68.113, 68.155],
[ 67.81 , 67.865, 67.905, 67.93 , 67.939],
[ 67.915, 67.934, 67.936, 67.919, 67.875],
[ 67.929, 67.9 , 67.851, 67.756, 67.608],
[ 67.768, 67.653, 67.477, 57.116, 56.783]]],
[[[ 16.01 , 16.293, 16.561, 16.813, 17.047],
[ 16.255, 16.505, 16.736, 16.947, 17.133],
[ 16.794, 16.997, 17.174, 17.319, 17.42 ],
[ 17.108, 17.241, 17.324, 17.321, 7.2 ],
[ 17.07 , 17.015, 6.972, 6.323, 4.544]],
[[ 68.647, 68.741, 68.825, 68.899, 68.962],
[ 68.585, 68.658, 68.719, 68.769, 68.806],
[ 68.737, 68.782, 68.814, 68.834, 68.836],
[ 68.818, 68.829, 68.83 , 68.804, 68.761],
[ 68.768, 68.731, 68.682, 68.581, 68.431]]],
[[[ 16.769, 17.069, 17.356, 17.63 , 17.888],
[ 17.057, 17.329, 17.587, 17.829, 18.053],
[ 17.654, 17.89 , 18.108, 18.305, 18.477],
[ 18.057, 18.248, 18.412, 18.545, 18.634],
[ 18.194, 18.316, 18.389, 18.375, 8.243]],
[[ 69.355, 69.461, 69.559, 69.647, 69.725],
[ 69.323, 69.41 , 69.488, 69.555, 69.613],
[ 69.511, 69.575, 69.628, 69.672, 69.704],
[ 69.64 , 69.677, 69.708, 69.719, 69.722],
[ 69.658, 69.66 , 69.661, 69.63 , 69.584]]]]])
I am trying to combine multiple JSON files in a Ubuntu platform. As an example, the data from two files are as follows:
File_1
{
"artist":"Gob",
"timestamp":"2011-08-09 01:59:41.352247",
"similars":[
[
"TRTOVWD128F92F4227",
1
],
[
"TRUXNUD128F92F41D0",
0.97294099999999994
],
[
"TRNNOJO128F42992E9",
0.073926900000000004
],
[
"TRGZHTT128F423B2A4",
0.068387699999999996
],
[
"TRGYKYD128F42625F6",
0.065579700000000005
],
[
"TRGIWHY128F42625F5",
0.064063700000000001
],
[
"TRJCJTX128F930CACE",
0.063140100000000005
],
[
"TRMYNWT128F426254B",
0.0613825
],
[
"TRRQOJI128F428C865",
0.061121599999999998
],
[
"TRBNYHM128F428A569",
0.061121599999999998
],
[
"TRDLOYE128F4241E72",
0.060951900000000003
],
[
"TRNRVEW12903CBA24F",
0.060332700000000003
],
[
"TRKKIPG12903CBA083",
0.060155
],
[
"TRZHTGP128F428A63B",
0.059873599999999999
],
[
"TRKQSGZ128F428A851",
0.059873599999999999
],
[
"TRTOPDF128F42AD88A",
0.059687799999999999
],
[
"TRIWOPM128F4241E53",
0.058958900000000002
],
[
"TRCCJUW128F14652DB",
0.057935
],
[
"TRERDDF128F428ECC4",
0.057566600000000002
],
[
"TROKWNN128F421A3D8",
0.057379800000000002
],
[
"TRWGOOK128F42AE765",
0.057125000000000002
],
[
"TRFMNKP128F428ADC0",
0.056875099999999998
],
[
"TRDMLZT128F42A01A8",
0.055808900000000002
],
[
"TRGCJVM128E0780E48",
0.0547389
],
[
"TRRXGAY128F14652D7",
0.0538065
],
[
"TRIPEHH128F1462DFF",
0.052843000000000001
],
[
"TRDUOIP128F147D5A7",
0.051851500000000002
],
[
"TRZCHHD12903CC80A1",
0.051251699999999997
],
[
"TRFDDQS128F426243F",
0.051018300000000003
],
[
"TRZDKAR128F42591B8",
0.050740899999999999
],
[
"TRDVXUG128F1456CBF",
0.050486299999999998
],
[
"TRULRYN128F145FC1C",
0.050219800000000002
],
[
"TRMOWIA128F425CE0F",
0.049977500000000001
],
[
"TRUVPMZ128F42B6DF3",
0.049762000000000001
],
[
"TRSBDWW128F4262666",
0.049643699999999999
],
[
"TRKPHWQ128F4264F8C",
0.0495173
],
[
"TRBBLXU128F42623A1",
0.049416700000000001
],
[
"TRJKLLM128F1456C57",
0.049001599999999999
],
[
"TRSAAEI128F4216C24",
0.048813500000000003
],
[
"TRFXICT128F4264F8A",
0.048776199999999999
],
[
"TRINVLH12903CBE5A1",
0.048334500000000002
],
[
"TRMUUJR128F4262475",
0.048306500000000002
],
[
"TRTORTD128F1456AFA",
0.0468265
],
[
"TRECUJO12903CA7120",
0.046065599999999998
],
[
"TRXIRBQ128F93431BB",
0.0456938
],
[
"TRFDDVK128F42B6DF0",
0.045623799999999999
],
[
"TRSRGPM128F421A30B",
0.043976800000000003
],
[
"TRVUPPR128F429507D",
0.042872500000000001
],
[
"TRMHCZC128F428A4CD",
0.040675200000000002
],
[
"TRUFDRV128F4262352",
0.040675200000000002
],
[
"TRUZZHT128F93229AF",
0.039422199999999998
],
[
"TRLSIHL128F429AF18",
0.039002099999999998
],
[
"TRGETCK128F1460DB1",
0.038499499999999999
],
[
"TRSXXNU128F428AEF2",
0.038303799999999999
],
[
"TRFZXSY128F9330D9F",
0.037855199999999999
],
[
"TRPHFYF128F92F27FA",
0.037772100000000003
],
[
"TRNRHSL128F9337B55",
0.036998000000000003
],
[
"TRPTGNZ128F421A56B",
0.036713099999999999
],
[
"TRPAASI128F9337B6E",
0.036410499999999998
],
[
"TRGCROO128F93431C4",
0.035754300000000003
],
[
"TRCUHZL128F4235446",
0.034968699999999998
],
[
"TRDPOTJ128F429AF0C",
0.034860500000000003
],
[
"TROZUXM128F42790A2",
0.0346483
],
[
"TRJVLOQ128F9345A82",
0.034547799999999997
],
[
"TRQTFRP128F145FC1E",
0.033934600000000002
],
[
"TRQEWHR128F421A3F5",
0.032314599999999999
],
[
"TRNTPJA128F4265039",
0.030702900000000002
],
[
"TRDGXWY12903CF52BD",
0.030292300000000001
],
[
"TRBLEMZ128F93102D0",
0.029224300000000002
],
[
"TRBUUYO128F421A405",
0.028448500000000002
],
[
"TREVBDI12903CED7E6",
0.0279674
],
[
"TRKREBF128F429B317",
0.0258321
],
[
"TRZBYPR128F4233A8D",
0.025655000000000001
],
[
"TRTAZUQ12903CFEA78",
0.024545399999999998
],
[
"TRAIPRO128F429AE69",
0.024304699999999999
],
[
"TRTTVUZ128F92FADD3",
0.023320899999999999
],
[
"TRUYEJI128F4265041",
0.022173700000000001
],
[
"TRAXVGT128F9344507",
0.0213992
],
[
"TRJJBLH128F4260DA1",
0.0175365
],
[
"TRAMCWR128F4233F7F",
0.0161158
],
[
"TRXBLME128F424330F",
0.015760900000000001
],
[
"TRMUQXM128F4260D99",
0.015696000000000002
],
[
"TRHRZBJ128EF345514",
0.0156951
],
[
"TRJXIBT128F42454DB",
0.014519199999999999
],
[
"TRTHPOY128F9345AA5",
0.0137264
],
[
"TRRFGJU128F933B2E6",
0.0012336199999999999
],
[
"TRMYJUA128F428A590",
0.00123149
],
[
"TRNMVTE128F933B2EC",
0.00122703
],
[
"TRYALZM128F1483C7D",
0.0012245299999999999
],
[
"TRZVEJU128F4234F4E",
0.00121805
],
[
"TRQAZDO128F145639F",
0.0012166600000000001
],
[
"TRJXNJM12903CF57ED",
0.0012155
],
[
"TRVAOGO128F427C9D6",
0.00120951
],
[
"TRZMZDS128F422843B",
0.0012065000000000001
],
[
"TRXIEOF12903CE8212",
0.0012058699999999999
],
[
"TRPVVUG128F42A36AA",
0.0012057599999999999
],
[
"TRXGVXS128F428AA5C",
0.0012019400000000001
],
[
"TRUBOGF128E078A5B9",
0.0012017900000000001
],
[
"TRITZSB128F4277CC2",
0.0012014
],
[
"TRGHPHX128F9343544",
0.0011975600000000001
],
[
"TRUKWPE128F428114F",
0.00119666
],
[
"TROBGRB128F93229AB",
0.0011964199999999999
],
[
"TRGKTMW12903CFAE65",
0.00119637
]
],
"tags":[
[
"punk rock",
"100"
],
[
"punk",
"60"
]
],
"track_id":"TRAAAFD128F92F423A",
"title":"Face the Ashes"
}
File_2
{
"artist":"CLP",
"timestamp":"2011-08-02 06:36:59.879759",
"similars":[
],
"tags":[
],
"track_id":"TRAAAVG12903CFA543",
"title":"Insatiable (Instrumental Version)"
}
I wrote a Python script to combine them. I added a new line and a comma after each record.
import glob
read_files = glob.glob("*.json")
with open("merged_file.json", "wb") as outfile:
for f in read_files:
with open(f, "rb") as infile:
outfile.write(infile.read())
outfile.write(',\n')
The output from the merge file is:
{
"artist":"Gob",
"timestamp":"2011-08-09 01:59:41.352247",
"similars":[
[
"TRTOVWD128F92F4227",
1
],
[
"TRUXNUD128F92F41D0",
0.97294099999999994
],
[
"TRNNOJO128F42992E9",
0.073926900000000004
],
[
"TRGZHTT128F423B2A4",
0.068387699999999996
],
[
"TRGYKYD128F42625F6",
0.065579700000000005
],
[
"TRGIWHY128F42625F5",
0.064063700000000001
],
[
"TRJCJTX128F930CACE",
0.063140100000000005
],
[
"TRMYNWT128F426254B",
0.0613825
],
[
"TRRQOJI128F428C865",
0.061121599999999998
],
[
"TRBNYHM128F428A569",
0.061121599999999998
],
[
"TRDLOYE128F4241E72",
0.060951900000000003
],
[
"TRNRVEW12903CBA24F",
0.060332700000000003
],
[
"TRKKIPG12903CBA083",
0.060155
],
[
"TRZHTGP128F428A63B",
0.059873599999999999
],
[
"TRKQSGZ128F428A851",
0.059873599999999999
],
[
"TRTOPDF128F42AD88A",
0.059687799999999999
],
[
"TRIWOPM128F4241E53",
0.058958900000000002
],
[
"TRCCJUW128F14652DB",
0.057935
],
[
"TRERDDF128F428ECC4",
0.057566600000000002
],
[
"TROKWNN128F421A3D8",
0.057379800000000002
],
[
"TRWGOOK128F42AE765",
0.057125000000000002
],
[
"TRFMNKP128F428ADC0",
0.056875099999999998
],
[
"TRDMLZT128F42A01A8",
0.055808900000000002
],
[
"TRGCJVM128E0780E48",
0.0547389
],
[
"TRRXGAY128F14652D7",
0.0538065
],
[
"TRIPEHH128F1462DFF",
0.052843000000000001
],
[
"TRDUOIP128F147D5A7",
0.051851500000000002
],
[
"TRZCHHD12903CC80A1",
0.051251699999999997
],
[
"TRFDDQS128F426243F",
0.051018300000000003
],
[
"TRZDKAR128F42591B8",
0.050740899999999999
],
[
"TRDVXUG128F1456CBF",
0.050486299999999998
],
[
"TRULRYN128F145FC1C",
0.050219800000000002
],
[
"TRMOWIA128F425CE0F",
0.049977500000000001
],
[
"TRUVPMZ128F42B6DF3",
0.049762000000000001
],
[
"TRSBDWW128F4262666",
0.049643699999999999
],
[
"TRKPHWQ128F4264F8C",
0.0495173
],
[
"TRBBLXU128F42623A1",
0.049416700000000001
],
[
"TRJKLLM128F1456C57",
0.049001599999999999
],
[
"TRSAAEI128F4216C24",
0.048813500000000003
],
[
"TRFXICT128F4264F8A",
0.048776199999999999
],
[
"TRINVLH12903CBE5A1",
0.048334500000000002
],
[
"TRMUUJR128F4262475",
0.048306500000000002
],
[
"TRTORTD128F1456AFA",
0.0468265
],
[
"TRECUJO12903CA7120",
0.046065599999999998
],
[
"TRXIRBQ128F93431BB",
0.0456938
],
[
"TRFDDVK128F42B6DF0",
0.045623799999999999
],
[
"TRSRGPM128F421A30B",
0.043976800000000003
],
[
"TRVUPPR128F429507D",
0.042872500000000001
],
[
"TRMHCZC128F428A4CD",
0.040675200000000002
],
[
"TRUFDRV128F4262352",
0.040675200000000002
],
[
"TRUZZHT128F93229AF",
0.039422199999999998
],
[
"TRLSIHL128F429AF18",
0.039002099999999998
],
[
"TRGETCK128F1460DB1",
0.038499499999999999
],
[
"TRSXXNU128F428AEF2",
0.038303799999999999
],
[
"TRFZXSY128F9330D9F",
0.037855199999999999
],
[
"TRPHFYF128F92F27FA",
0.037772100000000003
],
[
"TRNRHSL128F9337B55",
0.036998000000000003
],
[
"TRPTGNZ128F421A56B",
0.036713099999999999
],
[
"TRPAASI128F9337B6E",
0.036410499999999998
],
[
"TRGCROO128F93431C4",
0.035754300000000003
],
[
"TRCUHZL128F4235446",
0.034968699999999998
],
[
"TRDPOTJ128F429AF0C",
0.034860500000000003
],
[
"TROZUXM128F42790A2",
0.0346483
],
[
"TRJVLOQ128F9345A82",
0.034547799999999997
],
[
"TRQTFRP128F145FC1E",
0.033934600000000002
],
[
"TRQEWHR128F421A3F5",
0.032314599999999999
],
[
"TRNTPJA128F4265039",
0.030702900000000002
],
[
"TRDGXWY12903CF52BD",
0.030292300000000001
],
[
"TRBLEMZ128F93102D0",
0.029224300000000002
],
[
"TRBUUYO128F421A405",
0.028448500000000002
],
[
"TREVBDI12903CED7E6",
0.0279674
],
[
"TRKREBF128F429B317",
0.0258321
],
[
"TRZBYPR128F4233A8D",
0.025655000000000001
],
[
"TRTAZUQ12903CFEA78",
0.024545399999999998
],
[
"TRAIPRO128F429AE69",
0.024304699999999999
],
[
"TRTTVUZ128F92FADD3",
0.023320899999999999
],
[
"TRUYEJI128F4265041",
0.022173700000000001
],
[
"TRAXVGT128F9344507",
0.0213992
],
[
"TRJJBLH128F4260DA1",
0.0175365
],
[
"TRAMCWR128F4233F7F",
0.0161158
],
[
"TRXBLME128F424330F",
0.015760900000000001
],
[
"TRMUQXM128F4260D99",
0.015696000000000002
],
[
"TRHRZBJ128EF345514",
0.0156951
],
[
"TRJXIBT128F42454DB",
0.014519199999999999
],
[
"TRTHPOY128F9345AA5",
0.0137264
],
[
"TRRFGJU128F933B2E6",
0.0012336199999999999
],
[
"TRMYJUA128F428A590",
0.00123149
],
[
"TRNMVTE128F933B2EC",
0.00122703
],
[
"TRYALZM128F1483C7D",
0.0012245299999999999
],
[
"TRZVEJU128F4234F4E",
0.00121805
],
[
"TRQAZDO128F145639F",
0.0012166600000000001
],
[
"TRJXNJM12903CF57ED",
0.0012155
],
[
"TRVAOGO128F427C9D6",
0.00120951
],
[
"TRZMZDS128F422843B",
0.0012065000000000001
],
[
"TRXIEOF12903CE8212",
0.0012058699999999999
],
[
"TRPVVUG128F42A36AA",
0.0012057599999999999
],
[
"TRXGVXS128F428AA5C",
0.0012019400000000001
],
[
"TRUBOGF128E078A5B9",
0.0012017900000000001
],
[
"TRITZSB128F4277CC2",
0.0012014
],
[
"TRGHPHX128F9343544",
0.0011975600000000001
],
[
"TRUKWPE128F428114F",
0.00119666
],
[
"TROBGRB128F93229AB",
0.0011964199999999999
],
[
"TRGKTMW12903CFAE65",
0.00119637
]
],
"tags":[
[
"punk rock",
"100"
],
[
"punk",
"60"
]
],
"track_id":"TRAAAFD128F92F423A",
"title":"Face the Ashes"
},
{
"artist":"CLP",
"timestamp":"2011-08-02 06:36:59.879759",
"similars":[
],
"tags":[
],
"track_id":"TRAAAVG12903CFA543",
"title":"Insatiable (Instrumental Version)"
}
When I validate these records using JSON Lint (http://jsonlint.com/), it tells me that the file is broken and not a valid JSON. even after spending quite sometime, I am not able to figure out what is going wrong with the merging. It would be helpful is anyone has any thoughts on this.
You can't just concatenate two JSON strings to make valid JSON (or combine them by tacking ',\n' to the end of each).
Instead, you could combine the two (as Python objects) into a Python list, then use json.dump to write it to a file as JSON:
import json
import glob
result = []
for f in glob.glob("*.json"):
with open(f, "rb") as infile:
result.append(json.load(infile))
with open("merged_file.json", "wb") as outfile:
json.dump(result, outfile)
If you wanted to do it without the (unnecesssary) intermediate step of parsing each JSON file, you could merge them into a list like this:
import glob
read_files = glob.glob("*.json")
with open("merged_file.json", "wb") as outfile:
outfile.write('[{}]'.format(
','.join([open(f, "rb").read() for f in read_files])))
There is a module called jsonmerge which merges dictionaries. It can be used very simple by just providing two dictionaries or you can define schema's that described how to merge, like instead of overwriting same key's, automatically create a list and append to it.
base = {
"foo": 1,
"bar": [ "one" ],
}
head = {
"bar": [ "two" ],
"baz": "Hello, world!"
}
from jsonmerge import merge
result = merge(base, head)
print(result)
>>> {'foo': 1, 'bar': ['two'], 'baz': 'Hello, world!'}
More examples with complex rules: https://pypi.org/project/jsonmerge/#description
If you wanted to produce a JSON list of those objects, you are missing opening [ and closing ] brackets here, and will have written one comma too many.
It'll be easier to have Python decode the objects, then write the output as a new JSON list:
import json
import glob
read_files = glob.glob("*.json")
output_list = []
for f in read_files:
with open(f, "rb") as infile:
output_list.append(json.load(infile))
with open("merged_file.json", "wb") as outfile:
json.dump(output_list, outfile)
Python solution:
You have file1.json and file2.json files.
Each file has structure:
[{"key1": "value1"}] - (in file1)
[{"key2": "value2"}] - (in file2)
And your goal to merge them and get next view:
[{"key1": "value1"},
{"key2": "value2"}]
Code:
import glob
glob_data = []
for file in glob.glob('../../file*.json'):
with open(file) as json_file:
data = json.load(json_file)
i = 0
while i < len(data):
glob_data.append(data[i])
i += 1
with open('../../finalFile.json', 'w') as f:
json.dump(glob_data, f, indent=4)
Your generated JSON isn't valid. You end up with something like:
{...}, {...}
which, of course, is not valid.
If you want to go with your approach, you should remove the last } and the first { from the new file, and then merge them (you still have to be sure that the JSON files you want to merge are in a valid format).
I suggest I use the json module for what you need to achieve, it is cleaner.