I'm currently working with JSON API data which updated daily by my data team. I only need to extract certain information from the JSON file using pandas. So I managed to transform the data by pulling 1 sample JSON file from the API provided. However, I also need to update the transformed data in pandas with other JSON files from the API and I'm stuck here. FYI the JSON files from the API contain in folders name by according to date pulled
The following is the code I used to pull 1 sample JSON data for the transformation process.
#pull 1 sample data from API
df = requests.get('http://phpweb.123.co/atz-pull/output/2019-07-30/abc123.json').json()
So I managed to transform the data into the form i required. However I need to update the transformed data with other json files from different date folders.
Let say the output folders from the api also consist of the following (note that the json file name is unique ;
/output/2019-07-30/abc456.json
/output/2019-07-30/def678.json
/output/2019-07-29/wef678.json
Appreciate helps
Related
The values for data.CustomerNumber do not get written to the file, the other elements do.
The CustomerNumber is Pascal case because it is a custom value in the system I am pulling the data out of.
I have tried the map complex values option, advanced editor, deleted, started over. I originally was trying to save this to a Azure Sql table but gave up , and just tried to write to a file. I have been able to do this type of mapping in other pipelines so I am really at a loss for why it doesn't work in this case.
The actual json content for the mapping
In data factory I'm using a Copy Data activity where the Source is a REST API, that's returning data in JSON format, with a Sink thats a JSON file type in ADLS Storage.
The problem I'm having is that only the first record in the "Domestic Title Nos" array appears in the JSON file. The remaining records, however many, are not to be seen.
Also, when I "Preview" my data in the Source page, it shows all the nested data in the array.
API Call Returns this JSON data
Mapping in my Copy Data activity
File contents after sink to JSON file type
As shown in below screenshot, you have used items as Collection reference.
You need to use “Domestic Title Nos” as Collection reference.
Note: For records where the array marked as collection reference is
empty and the check box is selected, the entire record is skipped.
Refer - Schema and data type mapping in copy activity
In the end it worked out to be easier to download the entire JSON document using
this mapping and then used "Flatten" in a Data Flow to extract the parts that I needed.
I am a beginner in angular. My question is - Based on the excel file which user uploads, api returns some data in JSON format. So the column headers and data are dynamic. I want to display the json data in HTML table.
I want to know how should my json look like and its respective TS and HTML code.
User upload an excel file like this
file example
My output should look like this :
output
First column and row is dynamic.
As you are new to Angular, I will show you one of the procedure you can use
Angular part:
Every time the server sends data, you can get your data by HTTP methods or web-socket by "service" from angular.
Service:
in services, you can import HTTP to get data and after you get data you can emit it by using "behavior-subject" to any component you want which contains your tables and save data values there. (behavior-subject is from rxjs library).
Component:
Now your component in angular has the json data every time and you can analyze
your JSON file, based on your JSON shape you can use *ngfor in the HTML part of the component table and it will change the table online.
Hi is it possible to import any random json file into cassandra.
The json file is not exported from sstable2json. The json file is from a different website and needs to be imported into cassandra. Please could anyone advise whether this is possible
JSON support won't be introduced until Cassandra 3.0 (see CASSANDRA-7970) and in this case you still need to define a schema for your json data to map to. You do have some other options:
Use maps which sort of map to JSON. Maps can be indexed as of Cassandra 2.1 (CASSANDRA-4511) There is also a good Stack Exchange post about this.
You mention 'any random json file'. You could just have a string column that contains the raw JSON, but then you lose any query-ability of that data.
Come up with some kind of schema for your JSON data and map it to a CQL table and write some code that parses the JSON and writes it to the CQL table mapping to that data. This doesn't sound like an option for you since you want to be able to import any random JSON file.
If you are looking to only do json document storage, you might want to look at more document-oriented solutions instead of a column-oriented solution like cassandra.
Looking for the correct json formating for a Highstock multiple line graph:
Highstock is a great graphing api with lots of documentation. I just can't seem to find out how to format the json file. This is the graph that I am trying to set up:
http://www.highcharts.com/stock/demo/compare
This is the api document that discusses how to load data through json:
http://docs.highcharts.com/#preprocesssing-data-from-a-file
^--- Only problem here is that the json data example is set up for -Highcharts- not Highstock.
My json data follows this format. There should be six lines of data on the graph if it loads properly:
[timestamp, lineA, lineB, lineC, lineE, lineF]
Example:
[1366009207,-46.11,-19.71,-36.94,-20.21,-20.88,8.84]
[1366009217,-31.38,-21.74,-27.27,-24.64,-22.66,8.77]
etc...
Found the answer! Here is an example of their json data.
http://www.highcharts.com/samples/data/jsonp.php?filename=msft-c.json&callback=?
The key: Each graph-line must be loaded as a seperate json file.