I have tried to use Google Vision API Text detection feature and the web demo of Google to OCR my image. Two results is not same.
Firstly, i tried it with demo at url, https://cloud.google.com/vision/docs/drag-and-drop. Finally, i tried it with google api code by python language. Two results is not same and i don't know why . Could you please help me this problem?
My image: http://dfp.crawl.kyanon.digital/crawled_images/m.vta/1931/m.vta-home-slidebanner-image/2/assets/100000_samsung-galaxy-m20.png
My api result: "SAMSUNG Galaxy M20Siêu Pin vô doi, sac nhanh tuc thiMoiSAMSUNG4.990.000dTrà gop 0%Mua ngay"
My web demo result: https://imge.to/i/q4gRw
Thank you very much
my python code here:
client = vision.ImageAnnotatorClient()
raw_byte = cv2.imencode('.jpg', image)[1].tostring()
post_image = types.Image(content=raw_byte)
image_context = vision.types.ImageContext()
response = client.text_detection(image=post_image, image_context=image_context)
This is Typescript code.
But the idea is not to use text_detection but something like document_text_detection (unsure what the python API specifically provides).
Using documentTextDetection() instead of textDetection() solved the exact same problem for me.
const fs = require("fs");
const path = require("path");
const vision = require("#google-cloud/vision");
async function quickstart() {
let text = '';
const fileName = "j056vt-_800w_800h_sb.jpg";
const imageFile = fs.readFileSync(fileName);
const image = Buffer.from(imageFile).toString("base64");
const client = new vision.ImageAnnotatorClient();
const request = {
image: {
content: image
},
imageContext: {
languageHints: ["vi-VN"]
}
};
const [result] = await client.documentTextDetection(request);
// OUTPUT METHOD A
for (const tmp of result.textAnnotations) {
text += tmp.description + "\n";
}
console.log(text);
const out = path.basename(fileName, path.extname(fileName)) + ".txt";
fs.writeFileSync(out, text);
// OUTPUT METHOD B
const fullTextAnnotation = result.fullTextAnnotation;
console.log(`Full text: ${fullTextAnnotation.text}`);
fullTextAnnotation.pages.forEach(page => {
page.blocks.forEach(block => {
console.log(`Block confidence: ${block.confidence}`);
block.paragraphs.forEach(paragraph => {
console.log(`Paragraph confidence: ${paragraph.confidence}`);
paragraph.words.forEach(word => {
const wordText = word.symbols.map(s => s.text).join("");
console.log(`Word text: ${wordText}`);
console.log(`Word confidence: ${word.confidence}`);
word.symbols.forEach(symbol => {
console.log(`Symbol text: ${symbol.text}`);
console.log(`Symbol confidence: ${symbol.confidence}`);
});
});
});
});
});
}
quickstart();
Actually, comparing both of your results, the only difference I see is the way the results are displayed. The Google Cloud Drag and Drop site displays the results with the bounding boxes and tries to find areas of text.
The response you get with your python script includes the same information. A few examples:
texts = response.text_annotations
print([i.description for i in texts])
# prints all the words that were found in the image
print([i.bounding_poly.vertices for i in texts])
# prints all boxes around detected words
Feel free to ask more questions for clarification.
A few other thoughts:
Are you preprocessing your images?
What size are the images?
Related
we're using Google's Firestore for embedded machine configuration data. Because this data controls a configurable pageflow and lots of other things, it's segmented up into lots of subcollections. Each machine has it's own top level document in this system. However, it takes forever when we go to add machines to the fleet because we have to manually copy over all this data in multiple documents. Does anyone know how to go about recursively copying a Firestore document, all it's subcollections, their documents, subcollections, etc in Python. You'd have a document ref to the top level as well as a name for the new top level doc.
You can use something like this to recursively read and write from a collection to another one:
def read_recursive(
source: firestore.CollectionReference,
target: firestore.CollectionReference,
batch: firestore.WriteBatch,
) -> None:
global batch_nr
for source_doc_ref in source:
document_data = source_doc_ref.get().to_dict()
target_doc_ref = target.document(source_doc_ref.id)
if batch_nr == 500:
log.info("commiting %s batched operations..." % batch_nr)
batch.commit()
batch_nr = 0
batch.set(
reference=target_doc_ref,
document_data=document_data,
merge=False,
)
batch_nr += 1
for source_coll_ref in source_doc_ref.collections():
target_coll_ref = target_doc_ref.collection(source_coll_ref.id)
read_recursive(
source=source_coll_ref.list_documents(),
target=target_coll_ref,
batch=batch,
)
batch = db_client.batch()
read_recursive(
source=db_client.collection("src_collection_name"),
target=db_client.collection("target_collection_name"),
batch=batch,
)
batch.commit()
Writes are in batches and this saves a lot of time (in my case it finished in half the time compared with set).
The questions asks for Python, but in my case I needed to do recursive deep copy of Firestore docs / collections in NodeJS (Typescript), and using a Document as starting point of the recursion.
(This is a solution based on the Python script by #cristi)
Function definition
import {
CollectionReference,
DocumentReference,
DocumentSnapshot,
QueryDocumentSnapshot,
WriteBatch,
} from 'firebase-admin/firestore';
interface FirestoreCopyRecursiveContext {
batchSize: number;
/**
* Wrapped Firestore WriteBatch. In firebase-admin#11.0.1, you can't continue
* using the WriteBatch object after you call WriteBatch.commit().
*
* Hence, we need to replaced "used up" WriteBatch's with new ones.
* We also need to reset the count after committing, and because we
* want all recursive invocations to share the same count + WriteBatch instance,
* we pass this data via object reference.
*/
writeBatch: {
writeBatch: WriteBatch,
/** Num of items in current batch. Reset to 0 when `commitBatch` commits. */
count: number;
};
/**
* Function that commits the batch if it reached the limit or is forced to.
* The WriteBatch instance is automatically replaced with fresh one
* if commit did happen.
*/
commitBatch: (force?: boolean) => Promise<void>;
/** Callback to insert custom logic / write operations when we encounter a document */
onDocument?: (
sourceDoc: QueryDocumentSnapshot | DocumentSnapshot,
targetDocRef: DocumentReference,
context: FirestoreCopyRecursiveContext
) => unknown;
/** Callback to insert custom logic / write operations when we encounter a collection */
onCollection?: (
sourceDoc: CollectionReference,
targetDocRef: CollectionReference,
context: FirestoreCopyRecursiveContext
) => unknown;
logger?: Console['info'];
}
type FirestoreCopyRecursiveOptions = Partial<Omit<FirestoreCopyRecursiveContext, 'commitBatch'>>;
/**
* Copy all data from one document to another, including
* all subcollections and documents within them, etc.
*/
export const firestoreCopyDocRecursive = async (
/** Source Firestore Document Snapshot, descendants of which we want to copy */
sourceDoc: QueryDocumentSnapshot | DocumentSnapshot,
/** Destination Firestore Document Ref */
targetDocRef: DocumentReference,
options?: FirestoreCopyRecursiveOptions,
) => {
const batchSize = options?.batchSize ?? 500;
const writeBatchRef = options?.writeBatch || { writeBatch: firebaseFirestore.batch(), count: 0 };
const onDocument = options?.onDocument;
const onCollection = options?.onCollection;
const logger = options?.logger || console.info;
const commitBatch = async (force?: boolean) => {
// Commit batch only if size limit hit or forced
if (writeBatchRef.count < batchSize && !force) return;
logger(`Commiting ${writeBatchRef.count} batched operations...`);
await writeBatchRef.writeBatch.commit();
// Once we commit the batched data, we have to create another WriteBatch,
// otherwise we get error:
// "Cannot modify a WriteBatch that has been committed."
// See https://dev.to/wceolin/cannot-modify-a-writebatch-that-has-been-committed-265f
writeBatchRef.writeBatch = firebaseFirestore.batch();
writeBatchRef.count = 0;
};
const context = {
batchSize,
writeBatch: writeBatchRef,
onDocument,
onCollection,
commitBatch,
};
// Copy the contents of the current docs
const sourceDocData = sourceDoc.data();
await writeBatchRef.writeBatch.set(targetDocRef, sourceDocData, { merge: false });
writeBatchRef.count += 1;
await commitBatch();
// Allow to make additional changes to the target document from
// outside the func after copy command is enqueued / commited.
await onDocument?.(sourceDoc, targetDocRef, context);
// And try to commit in case user updated the count but forgot to commit
await commitBatch();
// Check for subcollections and docs within them
for (const sourceSubcoll of await sourceDoc.ref.listCollections()) {
const targetSubcoll = targetDocRef.collection(sourceSubcoll.id);
// Allow to make additional changes to the target collection from
// outside the func after copy command is enqueued / commited.
await onCollection?.(sourceSubcoll, targetSubcoll, context);
// And try to commit in case user updated the count but forgot to commit
await commitBatch();
for (const sourceSubcollDoc of (await sourceSubcoll.get()).docs) {
const targetSubcollDocRef = targetSubcoll.doc(sourceSubcollDoc.id);
await firestoreCopyDocRecursive(sourceSubcollDoc, targetSubcollDocRef, context);
}
}
// Commit all remaining operations
return commitBatch(true);
};
How to use it
const sourceDocRef = getYourFaveFirestoreDocRef(x);
const sourceDoc = await sourceDocRef.get();
const targetDocRef = getYourFaveFirestoreDocRef(y);
// Copy firestore resources
await firestoreCopyDocRecursive(sourceDoc, targetDocRef, {
logger,
// Note: In my case some docs had their doc ID also copied as a field.
// Because the copied documents get a new doc ID, we need to update
// those fields too.
onDocument: async (sourceDoc, targetDocRef, context) => {
const someDocPattern = /^nameOfCollection\/[^/]+?$/;
const subcollDocPattern = /^nameOfCollection\/[^/]+?\/nameOfSubcoll\/[^/]+?$/;
// Update the field that holds the document ID
if (targetDocRef.path.match(someDocPattern)) {
const docId = targetDocRef.id;
context.writeBatch.writeBatch.set(targetDocRef, { docId }, { merge: true });
context.writeBatch.count += 1;
await context.commitBatch();
return;
}
// In a subcollection, I had to update multiple ID fields
if (targetDocRef.path.match(subcollDocPattern)) {
const docId = targetDocRef.parent.parent?.id;
const subcolDocId = targetDocRef.id;
context.writeBatch.writeBatch.set(targetDocRef, { docId, subcolDocId }, { merge: true });
context.writeBatch.count += 1;
await context.commitBatch();
return;
}
},
});
I'm invoking a python script from node js. The python script retrieves data from a REST API and stores it in a dataframe and then there's a search function based on user input. I'm confused as to what variable type does python send the data to node js in? I've tried to convert into a string but in node js it says it is an unresolved variable type. Here's the code:
r = requests.get(url)
data = r.json()
nested = json.loads(r.text)
nested_full = json_normalize(nested)
req_data= json_normalize(nested,record_path ='items')
search = req_data.get(["name", "id"," ])
#search.head(10)
filter = sys.argv[1:]
print(filter)
input = filter[0]
print(input)
result = search[search["requestor_name"].str.contains(input)]
result = result.to_String(index=false)
response = '```' + str(result) + '```'
print(response)
sys.stdout.flush()
Here's the node js program that invokes the above python script. How do i store the output in a format which i can pass to another function in node?
var input = 'robert';
var childProcess = require("child_process").spawn('python', ['./search.py', input], {stdio: 'inherit'})
const stream = require('stream');
const format = require('string-format')
childProcess.on('data', function(data){
process.stdout.write("python script output",data)
result += String(data);
console.log("Here it is", data);
});
childProcess.on('close', function(code) {
if ( code === 1 ){
process.stderr.write("error occured",code);
process.exit(1);
}
else{
process.stdout.write('done');
}
});
According to the docs:
childProcess.stdout.on('data', (data) => {
console.log(`stdout: ${data}`);
});
I have followed the usage as per link below:
https://www.npmjs.com/package/react-native-paytm
This is my checksum generation code:
import Checksum
# initialize a dictionary
paytmParams = dict()
# put checksum parameters in Dictionary
paytmParams["MID"] = "*****************"
paytmParams["ORDER_ID"] = 'ORD001'
paytmParams["CUST_ID"] = 'CUST001'
paytmParams["INDUSTRY_TYPE_ID"] = 'Retail'
paytmParams["CHANNEL_ID"] = 'WAP'
paytmParams["TXN_AMOUNT"] = '1.00'
paytmParams["WEBSITE"] = 'WEBSTAGING'
paytmParams["EMAIL"] = '**************'
paytmParams["MOBILE_NO"] = '****************'
paytmParams["CALLBACK_URL"] = 'https://securegw-
stage.paytm.in/theia/paytmCallback?ORDER_ID=ORD001'
# Find your Merchant Key in your Paytm Dashboard at
https://dashboard.paytm.com/next/apikeys
checksum = Checksum.generate_checksum(paytmParams, "*************")
Code for react native:
import paytm from 'react-native-paytm';
import { Platform, DeviceEventEmitter, NativeModules,NativeEventEmitter} from 'react-native';
const paytmConfig = {
MID: '************',
WEBSITE: 'WEBSTAGING',
CHANNEL_ID: 'WAP',
INDUSTRY_TYPE_ID: 'Retail',
CALLBACK_URL: 'https://securegw-
stage.paytm.in/theia/paytmCallback?ORDER_ID=ORD001'
}
onPayTmResponse(response) {
// Process Response
// response.response in case of iOS
// reponse in case of Android
console.log(response);
}
runTransaction() {
const callbackUrl = 'https://securegw-
stage.paytm.in/theia/paytmCallback?ORDER_ID=ORD001'
const details = {
mode: 'Staging', // 'Staging' or 'Production'
mid: paytmConfig.MID,
industryType: paytmConfig.INDUSTRY_TYPE_ID,
website: paytmConfig.WEBSITE,
channel: paytmConfig.CHANNEL_ID,
amount: '1.00', // String
orderId: 'ORD001', // String
custId: 'CUST001', // String
email: '*****************', // String
phone: '***********', // S
checksumhash: '***********************************************', //From your server using PayTM Checksum Utility
callback: callbackUrl
};
paytm.startPayment(details);
}
The issue is that i cant even spot the error point here as there is no console. I could use some direction here. Need this for staging right now
in my cash use this npm #philly25/react-native-paytm and solved this error
are you sure onPaytmResponse is a paytm listener
like this:
componentWillMount() {
Paytm.addListener(Paytm.Events.PAYTM_RESPONSE, this.onPayTmResponse)
};
componentWillUnmount() {
Paytm.removeListener(Paytm.Events.PAYTM_RESPONSE, this.onPayTmResponse);
};
I'm trying to retrieve data programmatically through websockets and am failing due to my limited knowledge around this. On visiting the site at https://www.tradingview.com/chart/?symbol=ASX:RIO I notice one of the websocket messages being sent out is ~m~60~m~{"m":"quote_fast_symbols","p":["qs_p089dyse9tcu","ASX:RIO"]}
My code is as follows:
from websocket import create_connection
import json
ws = create_connection("wss://data.tradingview.com/socket.io/websocket?from=chart%2Fg0l68xay%2F&date=2019_05_27-12_19")
ws.send(json.dumps({"m":"quote_fast_symbols","p"["qs_p089dyse9tcu","ASX:RIO"]}))
result = ws.recv()
print(result)
ws.close()
Result of the print:
~m~302~m~{"session_id":"<0.25981.2547>_nyc2-charts-3-webchart-5#nyc2-compute-3_x","timestamp":1558976872,"release":"registry:5000/tvbs_release/webchart:release_201-106","studies_metadata_hash":"888cd442d24cef23a176f3b4584ebf48285fc1cd","protocol":"json","javastudies":"javastudies-3.44_955","auth_scheme_vsn":2}
I get this result no matter what message I send out, out of the almost multitude of messages that seem to be sent out. I was hoping one of the messages sent back will be the prices info for the low and highs for RIO. Is there other steps I should include to get this data? I understand there might be some form of authorisation needed but I dont know the workflow.
Yes, there is much more to setup and it needs to be done in order. The following example written in Node.js will subscribe to the BINANCE:BTCUSDT real time data and fetch historical 5000 bars on the daily chart.
Ensure you have proper value of the origin field set in header section before connecting. Otherwise your connection request will be rejected by the proxy. I most common ws there is no way to do this. Use faye-websocket instead
const WebSocket = require('faye-websocket')
const ws = new WebSocket.Client('wss://data.tradingview.com/socket.io/websocket', [], {
headers: { 'Origin': 'https://data.tradingview.com' }
});
After connecting you need to setup your data stream. I don't know if all of this commands needs to be performed. This probably can be shrink even more but it works. Basically what you need to do is to create new quote and chart sessions and within these sessions request stream of the data of the prior resolved symbol.
ws.on('open', () => {
const quote_session = 'qs_' + getRandomToken()
const chart_session = 'cs_' + getRandomToken()
const symbol = 'BINANCE:BTCUSDT'
const timeframe = '1D'
const bars = 5000
sendMsg(ws, "set_auth_token", ["unauthorized_user_token"])
sendMsg(ws, "chart_create_session", [chart_session, ""])
sendMsg(ws, "quote_create_session", [quote_session])
sendMsg(ws, "quote_set_fields", [quote_session,"ch","chp","current_session","description","local_description","language","exchange","fractional","is_tradable","lp","lp_time","minmov","minmove2","original_name","pricescale","pro_name","short_name","type","update_mode","volume","currency_code","rchp","rtc"])
sendMsg(ws, "quote_add_symbols",[quote_session, symbol, {"flags":['force_permission']}])
sendMsg(ws, "quote_fast_symbols", [quote_session, symbol])
sendMsg(ws, "resolve_symbol", [chart_session,"symbol_1","={\"symbol\":\""+symbol+"\",\"adjustment\":\"splits\",\"session\":\"extended\"}"])
sendMsg(ws, "create_series", [chart_session, "s1", "s1", "symbol_1", timeframe, bars])
});
ws.on('message', (msg) => { console.log(`RX: ${msg.data}`) })
And finally implementation of the helper methods
const getRandomToken = (stringLength=12) => {
characters = 'abcdefghijklmnopqrstuvwxyz0123456789'
const charactersLength = characters.length;
let result = ''
for ( var i = 0; i < stringLength; i++ ) {
result += characters.charAt(Math.floor(Math.random() * charactersLength))
}
return result
}
const createMsg = (msg_name, paramsList) => {
const msg_str = JSON.stringify({ m: msg_name, p: paramsList })
return `~m~${msg_str.length}~m~${msg_str}`
}
const sendMsg = (ws, msg_name, paramsList) => {
const msg = createMsg(msg_name, paramsList)
console.log(`TX: ${msg}`)
ws.send(createMsg(msg_name, paramsList))
}
i am trying to load an efi application from another efi application using loadimage and startimage protocols. but loadimage is getting succeeded,startimage failing with return value -1/0xffffffff. it would be very helpful if any one suggest some ideas, why it is failing. if there is any mistake in code please correct it.
EFI_STATUS LoadPythonBinary()
{
EFI_STATUS Status;
UINTN NumberOfFSHandles;
EFI_HANDLE *FSHandles;
EFI_GUID SimpleFileSystemGuid = EFI_SIMPLE_FILE_SYSTEM_PROTOCOL_GUID;
UINTN Index = 0;
EFI_BLOCK_IO *BlkIo;
EFI_SIMPLE_FILE_SYSTEM_PROTOCOL *FileSysProtocol = NULL;
EFI_DEVICE_PATH_PROTOCOL *FilePath;
EFI_HANDLE ImageHandle2 = NULL;
// EFI_DEVICE_PATH_PROTOCOL *DevicePath;
// EFI_HANDLE DeviceHandle;
EFI_HANDLE Controller=NULL;
EFI_LOADED_IMAGE_PROTOCOL *ImageInfo;
EFI_GUID EfiDevicePathProtocolGuid = EFI_DEVICE_PATH_PROTOCOL_GUID;
EFI_GUID EfiBlockIoProtocolGuid = EFI_BLOCK_IO_PROTOCOL_GUID;
const CHAR16 *FileName = L"Python.efi";
EFI_GUID EfiLoadedImageProtocol = EFI_LOADED_IMAGE_PROTOCOL_GUID;
// EFI_LOADED_IMAGE_PROTOCOL *LoadedImage;
char temp[MAX_PATH];
CHAR16 CmdLineParams[MAX_PATH] = L"fs0:\\GK\\Temp\\UnzipBuildTools.py fs0:\\GK\\Temp\\EFI.zip fs0:\\Test";
strcpy(temp,(const char *)StrDup16to8(CmdLineParams));
Status = gBS->LocateHandleBuffer(ByProtocol, &SimpleFileSystemGuid,NULL, &NumberOfFSHandles, &FSHandles);
if(!EFI_ERROR(Status))
{
for(Index = 0; Index < NumberOfFSHandles; Index++)
{
Status = gBS->HandleProtocol(FSHandles[Index], &SimpleFileSystemGuid, &BlkIo);
if(!EFI_ERROR(Status))
{
FilePath = FileDevicePath(FSHandles[Index],FileName);
Status = gBS->LoadImage(TRUE, gImageHandle, FilePath, NULL, 0, &ImageHandle2);
printf("Load Image Status = %x", Status);
if(!EFI_ERROR(Status))
{
printf("Image Loaded Successfully\n");
Status = gBS->HandleProtocol(ImageHandle2, &EfiLoadedImageProtocol,(VOID**)&ImageInfo);
if(!EFI_ERROR(Status))
{
if(ImageInfo->ImageCodeType == EfiLoaderCode)
{
gBS->FreePool(FilePath);
}
printf("Options :%s\n",temp);
printf("LoadedImage->ImageSize = %x", ImageInfo->ImageSize);
ImageInfo->LoadOptions = CmdLineParams;
ImageInfo->LoadOptionsSize = (UINT32)(wcslen(CmdLineParams));
ImageInfo->DeviceHandle = gImageHandle;
}
}
printf("About to start image\n");
Status = gBS->StartImage(ImageHandle2, NULL, NULL);
printf("StartImage Status = %x", Status);
if(!EFI_ERROR(Status))
{
printf("StartImage success\n");
break;
}
}
}
}
return Status;
}
Possible problem: Probably your target image (Python.efi) is not a valid UEFI application and can't be loaded by EFI_BOOT_SERVICES.StartImage() interface. For more information please consult the valid types of UEFI images loadable by UEFI boot service, check the session 7.4 in UEFI Spec 2.7.
Solution: Make sure that in the target application .inf file, the field MODULE_TYPE is configured with UEFI_APPLICATION and its .c file has the entry point signature to an UEFI application similar to:
EFI_STATUS
EFIAPI
MyEntryPointName (
IN EFI_HANDLE ImageHandle,
IN EFI_SYSTEM_TABLE *SystemTable
)
{
...
}
A functional code example can be consulted in LoadImageApp. This application load and start a target application named
HelloImageEntryPoint.efi with success.
-1 is not a valid EFI_STATUS if you are on a 64-bit system EFI_STATUS is 64-bit. Also if you use Print() %r will print out a string for EFI_STATUS.
The EFI_STATUS values returned from EFI services are define in the EFI Spec:
EFI_INVALID_PARAMETER - ImageHandle is either an invalid image handle or the image has already been initialized with StartImage
EFI_SECURITY_VIOLATION - The current platform policy specifies that the image should not be started.
Exit code from image - Exit code from image.
So did the code you loaded return an error back to you?