I want to make a request to an RESTapi of the Norwegian statistics bureau.
According to their api I need to make a POST request (thats not usual but that is ok) to get the tables that I need. In the body of the POST request I can specify with kind of table I want.
What I get is a JSON-STAT file. I need some help about how to handle it on Azure Data Factory. I read on documentation that ADF supports just JSON when using REST API. Does it mean that it also supports JSON-STAT ? If so How can I handle it on the activity ( source/sink ).
Thanks in advance,
Nunotrt
Here is the sample process of JSON-STAT file ingest from REST API . Create a new REST dataset. Create Linked services to the REST API. Provide REST dataset as the source for the copy activity and also with the POST request method, providing the request body({ "query": [ { "code": "Region", "selection": { "filter": "agg:KommSummer", "values": [ "K-3006" ] } } ], "response": { "format": "json-stat2" } })
Create a linked services for sink dataset. Create a file path for the sink dataset to get the file into the location. Result we get as per the expectations.
If you want to convert a json-stat file to csv or parquet, it is an alternative to you use a python code as a part of your Data Factory job.
The python library pyjstat makes it easy to convert json-stat to a DataFrame, and from DataFrame to any other file format.
Here are an example with json-stat 1.2
import pandas as pd
from pyjstat import pyjstat as pyj
# url
file_ssb = 'https://data.ssb.no/api/v0/dataset/1102.json'
# Upload data to memory
dataset = pyj.Dataset.read(file_ssb)
# Write json-stat to dataframe
df = dataset.write('dataframe')
print(df)
# Save as csv
df.to_csv('../Data/test_jsonstat.csv')
Related
I am not a software guy, but I am preparing a small script for delarative onboarding work using terraform. The declaration is in a json file and it is send to the device through terraform. But I do not want to keep the sensitive data like passwords and secret keys in the json file. I want to inject them from a seperate file in a vault to the json file when I am executing it in terraform. I can use terraform to read the data from a vault or a s3 bucket, but I am not sure how to add it to the json declaration. I saw in terraform there is a functions like jsondecode and jsonencode. But I am not sure whether they will be any use to my requirement.
Could someone help me on this please ?
Thansk.
Maybe this is something which could help you?
Create a data source which loads your template file, here you can pass in some vars.
some examples for variales:
data "template_file" "my_template" {
template = templatefile("template.json", {
foo = "bar"
another_foo = aws_s3_bucket.my_bucket.id
more_foo = var.my_variable
})
}
The template file could look like this:
{
"Value": "123",
"value_2": "${foo}",
"value_3": "${another_foo}",
"value_4": ${more_foo},
}
Good day to all,
I am new to survey 123 and argis online, but after loads of try, I ask for the help of the community.
I am trying to export data from the API rest of arcgis under CSV format by command line. As I have hundred of Feature Layer to export, I do not want to download it through the "Export Data" button
I have already download in json format through API rest, but the conversion of nested json into csv is tricky so I prefer to download the csv file from arcgis directly in command line.
For informations, the feature layer data that I want to export is from the results of survey made with 123survey.
Thus I have a table of data accessible to the following link : https://developers.arcgis.com/layers/6f8228f7bfa24a61a029aa9751da55d8/data/0
There is a tutorial in python to export csv from a layer, but in this tutorial the layer is a datasets, however I have a Feature Layer and not a datasets one :
https://developers.arcgis.com/labs/python/download-data/
The documentation to query a feature layer is available to the following link, but we can only get html or json object from the query :
https://developers.arcgis.com/labs/rest/query-a-feature-layer/
https://developers.arcgis.com/rest/services-reference/query-feature-service-layer-.htm
The documentation to export data is available to the following link :
https://developers.arcgis.com/rest/users-groups-and-items/export-item.htm
However I can not find the request to get the csv data from this table. I know how to do it through the web interface, but I want to know how to do it with requests.
I have already try the following requests with PostMan, but unfortunately none of the following have worked :
https://mycompany.maps.arcgis.com/home/rest/content/users/chargeetudeti/export?token=UsSMUWq.......&itemId=6f8228f7bfa24a61a029aa9751da55d8&exportFormat=CSV
https://mycompany.maps.arcgis.com/sharing/rest/content/users/chargeetudeti/export?token=UsSMUWq.......&itemId=6f8228f7bfa24a61a029aa9751da55d8&exportFormat=CSV
https://www.arcgis.com/sharing/rest/content/users/chargeetudeti/export?token=UsSMUWq.......&itemId=6f8228f7bfa24a61a029aa9751da55d8&exportFormat=CSV&exportParameters={"layers" : [ { "id" : 0 } ] }
https://services6.arcgis.com/xNifwnpI45tLCPU7/ArcGIS/rest/services/service_70df166a7c9649878cc260d81ee85e4e/FeatureServer/export?token=UsSMUWq.......&exportFormat=CSV&exportParameters={"layers" : [ { "id" : 0 } ] }
Do you have any clues ?
I made public the data to let you try to export the data into a csv format, from this feature layer : https://services6.arcgis.com/xNifwnpI45tLCPU7/arcgis/rest/services/service_70df166a7c9649878cc260d81ee85e4e/FeatureServer
I will be very gratefull if you could help me : )
If you think the title of this post or its tags, are not appropriate, please let me know.
Adrien
Within a web client using the Forge API, I would like to get a JSON array of Revit elements in a model. Using the Design Automation API, I am creating an Activity that uses FilteredElementCollector to retrieve the elements, but once I have the elements I'm not sure the best way to retrieve those results in my web service. This Activity does not need to write back to an .rvt file.
The CountItApp tutorial writes the results to a results.txt file in cloud storage, and the web app then downloads that results.txt file and parses the results. On my web client I want to display these results, but file I/O does not seem like a very good solution for JSON data. A couple alternatives I've considered:
Write results to an external database and query that database in my web application once the WorkItem completes. As far as I know this is not possible due to Forge's restrictions on network access within an Activity.
Pass the results with the onComplete callback. I don't know if this is possible.
Design automation allows you to post a workitem with output arguments with POST callback. This allows you to receive the output data as application/json if your output file generated by your activity is a json file.
Design automation also allows you to specify a variable workitem.id in your output url. When your workitem completes we shall call this url with the variable expanded to the id of that workitem. This dynamic variable path allows you to determine the workitem id associated with that callback.
Here is how you could go about. First define an activity with such an output parameter (verb: post) with a hardcoded local name result.json:
"results": {
"zip": false,
"ondemand": false,
"verb": "post",
"description": "Results",
"required": true,
"localName": "results.json"
}
In your appbundle code save the json contents in a file with hardcoded name result.json to the current working folder.
using (StreamWriter sw = File.CreateText("result.json"))
{
sw.WriteLine(JsonConvert.SerializeObject(data, Formatting.Indented));
sw.Close();
}
Then you can post a workitem like so:
"result": {
"verb": "post",
"url": "https://www.yourserver.com/results/$(workitem.id)"
}
In your server implementation of the callback you will get the json contents as the payload. You may read the results and communicate back to the client corresponding to the workitem id, using sockets or any other means of communication you may have with your client.
I am trying to make a post rest call to my service. My sample input json file is,
{
"$id": "1",
"description": "sfdasd"
}
I have one csv file which contain a bunch of id and description, So is there a option where I can convert csv file to json objects and pass them to post call?
Assuming your CSV file is called test.csv, located in JMeter's "bin" folder and looks like:
Add CSV Data Set Config to your Test Plan and configure it as follows:
You can inline the defined JMeter Variables directly into your request body like:
{
"$id": "${id}",
"description": "${description}"
}
So when you run the test the variables placeholders will automatically be substituted with the values from the CSV file in the HTTP Request sampler:
See Using CSV DATA SET CONFIG article for more information on JMeter tests parameterization using CSV files.
Json is just text. Send as is with the variable id taken from csv:
{ "${id}": "1", "description": "sfdasd" }
CSV data can be converted to JSON via a POJO using Jackson. If a POJO is not already defined or required, you can always use the Java Collection classes to store parsed data and later convert it to JSON. http://www.novixys.com/blog/convert-csv-json-java/ is a good reference on how to convert csv to java.
You can check the following links too.
1. directly convert CSV file to JSON file using the Jackson library
2. Converting an CSV file to a JSON object in Java
I'm working with JMeter to make some HTTP Requests to my Firebase Database. I am able to create json data with a regular request, as well as with a CSV file. I'm wondering if it's possible to update, or add to, a json object.
My json data looks something like what is below. Let's say I wanted to add a boolean node called "sold", to which I could make equal to true or false. Could I create it within that json object? If so, could I also make it so that only fields with a specific "name" get updated?
{
"Price": "5.00",
"name": "buyer#gmail.com",
"seller_name": "seller#gmail.com",
"time": 1496893589683,
}
Looking into Updating Data with PATCH chapter of Saving Data article you can update a single field using HTTP PATCH Method.
JMeter supports HTTP PATCH method since version 2.8 so you should be in a position to use it as well in your test.