I am using python 3 for functional testing of a bunch of rest endpoints.
But i cannot figure out the best way to validate the json reaponse ( verifying the type, required, missing and additional fields)
I thought of below options :
1. Writing custom code and validate the response while converting the data into python class objects.
2. Validate using json schema .
Option 1: would be difficult to maintain and need to add separate functions to all the data models.
Option 2 : i like it. But i dont want to write schema for each endpoint in separate file/object. Is there a way to put it in a single object like we have swagger yml file. That way would be easy to maintain.
I would like to know which option is the best and if there are other better options / libraries available.
I've been through the same process, but validating REST requests and responses with Java. In the end I went with JSON Schema (there's an equivalent Python implementation at https://pypi.python.org/pypi/jsonschema) because it was simple and powerful, and hand-crafting the validation for anything but a trivial payload soon became a nightmare. Also, reading a JSON Schema file is easier than reasoning about a long list of validation statements.
It's true you need to define the schema in a separate file, but this proved to be no big deal. And, if your endpoints share some common features you can modularise your schemas and reuse common parts. There's a good tutorial at Understanding JSON Schema.
Related
I want to use apache avro schema's for data serialization and deserialization.
I want to use it with json encoding.
I want to put several of this serialized objects using different schemas to the same "source" (it's a kafka topic).
When I read it back I have the need to be able to resolve the right schema for the current data entry.
But the serialized data don't have any schema information in it. And to test all possible schema's for compatibility (kind of a duck typing approach) would be pretty unclean and error prone (for data which fits to multiple schemas it would be unclear which one to take)
I'm currently thought about putting the namespace and object name inside the json data programatically. But such a solution would not belong to the avro standard and it would open a new error scenario where it's possible to put the wrong schema namespace and/or object name inside the data.
I'm wondering if there would be a better way. Or if I have a general flaw in my solution.
Background: I want to use it for kafka message but don't want to use the schema registry (don't wan't to have a new single point of failure). I also still want to have KSQL support which is only available for json format or for using avro with the schema registry)
I am creating a spring-boot application which will interact with elasticsearch using spring-data. But the problem is, my data in the elasticsearch is unpredictable. That means there can be slight changes in the fields like additional fields or can be totally new field coming in JSON. Please guide me for a solution to address that. Using normal repository is seems not working because I don't have a defined JSON format. Your guide will be highly appreciated.
You need to provide a bit more data on your case.
Normally, when you use #Field annotations, or introducing/dropping a new simple or object field, this should not be a problem at all since spring-data-elasticsearch updates mapping when you save to ElasticsearchRepository. In some cases, e.g. introducing a parent-child rel, you would need to drop and recreate index but this can also be done programatically, if needed.
If you need advanced mapping that is also changing dynamically, then you need to build and execute a mapping update request from your code on save (custom repo).
I've inherited some code which makes calls to a web API, and get's a deeply nested (up to eight levels) response.
I've written some code to flatten the structure so that it can be written to .csv files, and a SQL database, for people to consume more easily.
What I'd really like to do though is keep a version of the original response, so that there's a reference of the original structure if I ever want/need it.
I understand that HDF5 is primarily meant to store numerical data. Is there any reason not to use it to dump JSON blobs? It seems a lot easier than setting up a NoSQL database.
It should be fine. It sounds like you'd be storing each JSON response as a HDF5 variable length string. Which is fine, it's just a string to the library.
Do you plan to store each response as a separate dataset? That may be inefficient if you are talking about >1000's of responses.
Alternatively, you can create a 1-d extensible dataset, and just append to it with each response.
Decided it was easier to set up a Mongo database.
I am designing a system with 30,000 objects or so and can't decide between the two: either have a JSON file pre computed for each one and get data by pointing to URL of the file (I think Twitter does something similar) or have a PHP/Perl/whatever else script that will produce JSON object on the fly when requested, from let's say database, and send it back. Is one more suited for than another? I guess if it takes a long time to generate the JSON data it is better to have already done JSON files. What if generating is as quick as accessing a database? Although I suppose one has a dedicated table in the database specifically for that. Data doesn't change very often so updating is not a constant thing. In that respect the data is static for all intense and purposes.
Anyways, any thought would be much appreciated!
Alex
You might want to try MongoDB which retrieves the objects as JSON and is highly scalable and easy to setup.
I have an existing MySQL database, I would like to import the schema into Xcode and create a Core Data data model.
Is there a way (tool, process) to import the CREATE statements so I don't have to build the models "by hand"?
As an intermediary step I could convert to SQLite, I'm not worried about the relationships, foreign keys etc just auto-generating the Entities (Tables) and Properties (Columns).
Actually I needed the feature so badly too that I have decided to make an OSX utility to do so. BUT... then I found a utility in the Mac Appstore that (partially) solves this problem (it was free for some time, I do not know its current state). Its called JSONModeler and what it does is parsing a json tree and generates the coredata model and all derived NSManagedObject subclasses automatically. So a typical workflow would be:
Export the tables from MySQL to xml
Convert the xml to json
Feed the utility with that json and get your coredata model
Now, for a more complicated scenario (relationships etc) I guess you would have to tweak your xml so it would reflect a valid object tree. Then JSONModeler will be able to recreate that tree and export it for coredata.
The problem here is that entities are not tables and properties are not columns. Core Data is an object graph management system and not a database system. The difference is subtle but important. Core Data really doesn't have anything to do with SQL it just sometimes uses SQL as one its persistence options.
Core Data does use a proprietary sqlite schema and in principle you can duplicate that but I don't know of anyone who has succeeded in a robust manner except for very simple SQL databases. Even when they do, its a lot of work. Further, doing so is unsupported and the schema might break somewhere down the line.
The easiest and most robust solution is to write a utility app to read in the existing DB and create the object graph as it goes. You only have to run it once and you've got to create the data model anyway so it doesn't take much time.