I'm having problems to import JSON data into my angular application.
What I'm doing:
Import the json file into a variable:
import _datos from '../../../assets/data.json';
As specified in the answers of this question (How to import JSON File into a TypeScript file?), I'm using the typing.d.ts file in order to be able to read the information.
Declaring an interface in order to improve the readability of the code:
export interface Datos{
property1: String,
property2: String,
etc.
}
Declaring and instantiating the array with the json file imported:
datos: Datos[] = _datos;
What is expected
The variable "datos" should have the content of the JSON file as an array.
What actually happens
Type '{}' is missing the following properties from type 'Datos[]': length, pop, push, concat, and 26 more.ts(2740).
Further information
I've done it before with a smaller JSON file (this one has about 26 properties each object and +100k elements) and worked fine. However, this one is failing and I don't really know why.
Okay it was a very simple mistake:
One of the 26 properties in the JSON file was not written properly. In order for this to work, it is needed that the properties in the JSON file and the properties in the interface match.
The JSON file is something like this:
[
{
"organo_de_contratacion": "Consorcio de Transportes de la Bahía de Cádiz",
"estado_de_la_licitacion**:**": "Resuelta",
...
So that the : slipped before the ending of the property and missmatched the interface property.
To avoid the error:
Type '{}' is missing the following properties from type ...
Ensure that the object _datos is being assigned is an array of objects of type Datos.
This occurs when assigning an object { .. } to an array of objects [ {}, .. ]
Your variable that holds the imported array needs to be declared as follows:
_datos: Datos[] = [];
Then import your JSON file data into _datos.
I would suggest debugging the value of _datos before it is assigned to the datos variable.
Related
I am trying to code a file import module that takes CSV or XLSX files. I am stuck at the CSV import stage, trying to model the property that will hold the reader.
The problem arises because I want to be able to create a csv::Reader using either the ::from_path function (the way it will generally be used) or the ::from_reader function (mainly for tests).
The types returned by the two functions is different: Reader<File> and Reader<R> respectively.
The structures I have:
trait ReaderSeeker: Read + Seek {}
struct CsvReader {
header_map: Vec<(String, usize)>,
line_no: usize,
r: Box<csv::Reader<dyn ReaderSeeker>>,
sr: StringRecord,
}
Read and Seek are the traits that I use on the reader.
I get:
error[E0277]: the size for values of type (dyn ReaderSeeker + 'static) cannot be known at compilation time
I also tried:
struct CsvReader<R: Read + Seek> {
header_map: Vec<(String, usize)>,
line_no: usize,
r: Box<csv::Reader<R>>,
sr: StringRecord,
}
But that causes other issues when trying to instantiate the struct, as I need to pass the type then -which I do not want to do- as it is completely irrelevant to the caller.
How can I model this situation?
overall aim
I have data landing into blob storage from an azure service in form of json files where each line in a file is a nested json object. I want to process this with spark and finally store as a delta table with nested struct/map type columns which can later be queried downstream using the dot notation columnName.key
data nesting visualized
{
key1: value1
nestedType1: {
key1: value1
keyN: valueN
}
nestedType2: {
key1: value1
nestedKey: {
key1: value1
keyN: valueN
}
}
keyN: valueN
}
current approach and problem
I am not using the default spark json reader as it is resulting in some incorrect parsing of the files instead I am loading the files as text files and then parsing using udfs by using python's json module ( eg below ) post which I use explode and pivot to get the first level of keys into columns
#udf('MAP<STRING,STRING>' )
def get_key_val(x):
try:
return json.loads(x)
except:
return None
Post this initial transformation I now need to convert the nestedType columns to valid map types as well. Now since the initial function is returning map<string,string> the values in nestedType columns are not valid jsons so I cannot use json.loads, instead I have regex based string operations
#udf('MAP<STRING,STRING>' )
def convert_map(string):
try:
regex = re.compile(r"""\w+=.*?(?:(?=,(?!"))|(?=}))""")
obj = dict([(a.split('=')[0].strip(),(a.split('=')[1])) for a in regex.findall(s)])
return obj
except Exception as e:
return e
this is fine for second level of nesting but if I want to go further that would require another udf and subsequent complications.
question
How can I use a spark udf or native spark functions to parse the nested json data such that it is queryable in columnName.key format.
also there is no restriction of spark version, hopefully I was able to explain this properly. do let me know if you want me to put some sample data and the code for ease. Any help is appreciated.
I am creating a JSON model in UI5 from a json object. But it is missing the columns array in the JSON model. I checked the model in the console and the odata property had no columns. Check image for proof.
https://ibb.co/hLmYpZb
Hence I want to know how I can get the column to show up.
Here is my code.
var emptyModel = {
"columns": []
};
this.setModel(new JSONModel(JSON.stringify(emptyModel)), "selcontent");
I expect column to show up in the JSON model under the odata property.
JSONModel awaits a plain object or a URL string as the first argument.
Either the URL where to load the JSON from or a JS object (source)
The result of the stringified emptyModel is "{"columns":[]}" which is neither an object nor a URL.
I store a blob of Json in the datastore using JsonProperty.
I don't know the structure of the json data.
I am using endpoints proto datastore in order to retrieve my data.
The probleme is the json property is encoded in base64 and I want a plain json object.
For the example, the json data will be:
{
first: 1,
second: 2
}
My code looks something like:
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Model(EndpointsModel):
data = ndb.JsonProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class DataEndpoint(remote.Service):
#Model.method(path='mymodel2', http_method='POST',
name='mymodel.insert')
def MyModelInsert(self, my_model):
my_model.data = {"first": 1, "second": 2}
my_model.put()
return my_model
#Model.method(path='mymodel/{entityKey}',
http_method='GET',
name='mymodel.get')
def getMyModel(self, model):
print(model.data)
return model
API = endpoints.api_server([DataEndpoint])
When I call the api for getting a model, I get:
POST /_ah/api/myapi/v1/mymodel2
{
"data": "eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ=="
}
where eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ== is the base64 encoded of {"second": 2, "first": 1}
And the print statement give me: {u'second': 2, u'first': 1}
So, in the method, I can explore the json blob data as a python dict.
But, in the api call, the data is encoded in base64.
I expeted the api call to give me:
{
'data': {
'second': 2,
'first': 1
}
}
How can I get this result?
After the discussion in the comments of your question, let me share with you a sample code that you can use in order to store a JSON object in Datastore (it will be stored as a string), and later retrieve it in such a way that:
It will show as plain JSON after the API call.
You will be able to parse it again to a Python dict using eval.
I hope I understood correctly your issue, and this helps you with it.
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class MyApi(remote.Service):
# URL: .../_ah/api/myapi/v1/mymodel - POSTS A NEW ENTITY
#Sample.method(path='mymodel', http_method='GET', name='Sample.insert')
def MyModelInsert(self, my_model):
dict={'first':1, 'second':2}
dict_str=str(dict)
my_model.column1="Year"
my_model.column2=2018
my_model.column3=dict_str
my_model.put()
return my_model
# URL: .../_ah/api/myapi/v1/mymodel/{ID} - RETRIEVES AN ENTITY BY ITS ID
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
dict=eval(my_model.column3)
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
application = endpoints.api_server([MyApi], restricted=False)
I have tested this code using the development server, but it should work the same in production using App Engine with Endpoints and Datastore.
After querying the first endpoint, it will create a new Entity which you will be able to find in Datastore, and which contains a property column3 with your JSON data in string format:
Then, if you use the ID of that entity to retrieve it, in your browser it will show the string without any strange encoding, just plain JSON:
And in the console, you will be able to see that this string can be converted to a Python dict (or also a JSON, using the json module if you prefer):
I hope I have not missed any point of what you want to achieve, but I think all the most important points are covered with this code: a property being a JSON object, store it in Datastore, retrieve it in a readable format, and being able to use it again as JSON/dict.
Update:
I think you should have a look at the list of available Property Types yourself, in order to find which one fits your requirements better. However, as an additional note, I have done a quick test working with a StructuredProperty (a property inside another property), by adding these modifications to the code:
#Define the nested model (your JSON object)
class Structured(EndpointsModel):
first = ndb.IntegerProperty()
second = ndb.IntegerProperty()
#Here I added a new property for simplicity; remember, StackOverflow does not write code for you :)
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
column4 = ndb.StructuredProperty(Structured)
#Modify this endpoint definition to add a new property
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
#Add the new nested property here
dict=eval(my_model.column3)
my_model.column4=dict
print(json.dumps(my_model.column3))
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
With these changes, the response of the call to the endpoint looks like:
Now column4 is a JSON object itself (although it is not printed in-line, I do not think that should be a problem.
I hope this helps too. If this is not the exact behavior you want, maybe should play around with the Property Types available, but I do not think there is one type to which you can print a Python dict (or JSON object) without previously converting it to a String.
I'm using a service to load my form data into an array in my angular2 app.
The data is stored like this:
arr = []
arr.push({title:name})
When I do a console.log(arr), it is shown as Object. What I need is to see it
as [ { 'title':name } ]. How can I achieve that?
you may use below,
JSON.stringify({ data: arr}, null, 4);
this will nicely format your data with indentation.
To print out readable information. You can use console.table() which is much easier to read than JSON:
console.table(data);
This function takes one mandatory argument data, which must be an array or an object, and one additional optional parameter columns.
It logs data as a table. Each element in the array (or enumerable property if data is an object) will be a row in the table
Example:
first convert your JSON string to Object using .parse() method and then you can print it in console using console.table('parsed sring goes here').
e.g.
const data = JSON.parse(jsonString);
console.table(data);
Please try using the JSON Pipe operator in the HTML file. As the JSON info was needed only for debugging purposes, this method was suitable for me. Sample given below:
<p>{{arr | json}}</p>
You could log each element of the array separately
arr.forEach(function(e){console.log(e)});
Since your array has just one element, this is the same as logging {'title':name}
you can print any object
console.log(this.anyObject);
when you write
console.log('any object' + this.anyObject);
this will print
any object [object Object]