Reading key/value pairs with Google Cloud Vision + Natural Language = Possible? - ocr

Question

I'm currently doing a spike for a project and was hoping the community may be able to shed some light on things.
I would like to use Google Cloud Vision to scan the below image and then derive the key/value pairs from it (such as Title: Ground Rod..., Last Revision: June 27, 2012). This is a basic example, it could have much more data and the layout may be different to this.
Since there is no easy correlation between the key/values i'm not sure if this possible? Is it possible to train the google vision with example images? Or are there any other solutions that may be able to do this?
Thank you!

Answer 1

You can use the Cloud Vision API to scan the image and obtain the useful key/value pairs writing a program using the Vision Api Client Libraries. For example, dragging the image file here and switching to “Text” tab you can visualize this:
[...]DRAWING TITLE GROUND ROD STRUCTURAL STEEL CONNECTION DETAIL E-80-05 Division of Technical Resources Office of Research Facilities National Institutes of Health The formulae 5-steel- deal ** * -||-| S - for building H-KANA --- Ej as state of the art e A uto-aut - R4fco- biomedical research facilities: LAST REVISION JUNE 27, 2012
In “Document”, on block 10, you can read this:
G R O U N D R O D S T R U C T U R A L S T E E L C O N N E C T I O N D E T A I L
One last useful operation: open “JSON” tab, search “ground rod structural” in the navigator. If you go the fourth entrance and scroll up you will see the coordinates of the bounding boxes containing “June 27, 2012”, in reverse order: 2,1,0,2,etc. The 2 is defined as follows:
"boundingBox": {
"vertices": [
{
"x": 671,
"y": 1173
},
{
"x": 679,
"y": 1173
},
{
"x": 679,
"y": 1200
},
{
"x": 671,
"y": 1200
}
]
},
"text": "2",
"confidence": 0.96
}
],
"confidence": 0.98
}
],
"confidence": 0.99
}
],
"blockType": "TEXT",
"confidence": 0.99
}
]
}
],
As far as I know, footers for technical draws contain a well-structured limited type of information (as for example title, date and legislation rule in this case) that cannot change much.
Taking into account all the information gathered through the Cloud Vision API and the Client Libraries availability, a script could be written in one of the code languages to identify and save the useful blocks and post-process them to obtain key/value pairs. Found a document text detection sample here or a tutorial here.
It is not possible to train Cloud Vision API with example images. To train a Machine Learning model, it is required a training dataset with its corresponding answer, commonly denoted as target. You could use Cloud AI for Machine Learning purposes to do this.