Unable to insert data from object array or csv file into kusto table
My goal is to build a pipeline in Azure DevOps which reads data using PowerShell and writes the data into Kusto Table.
I was able to write the data which I have read from PowerShell to object Array or csv file but I am unable to figure out the ways in which this data can be inserted into Kusto table.
Could any one suggest the best way to write the data into kusto
one option would be to write your CSV payload to blob storage, then ingest that blob into your target table, by:
using a "queued ingestion" client in one of the client libraries: https://learn.microsoft.com/en-us/azure/kusto/api/
note that the .NET ingestion client library also provides you with methods to IngestFromStream or IngestFromDataReader, which handle writing the data to intermediate blob storage so that you don't have to
or by
issuing an .ingest command: https://learn.microsoft.com/en-us/azure/kusto/management/data-ingestion/ingest-from-storage. though using "direction ingestion" is less recommended for Production volumes
another option (not recommended for Production volume), would be using the .ingest inline (AKA "ingest push") option: https://learn.microsoft.com/en-us/azure/kusto/management/data-ingestion/ingest-inline
for example:
.create table sample_table (a:string, b:int, c:datetime)
.ingest inline into table sample_table <|
hello,17,2019-08-16 00:52:07
world,71,2019-08-16 00:52:08
"isn't, this neat?",-13,2019-08-16 00:52:09
which will append the above records to the table:
| a | b | c |
|-------------------|------|-----------------------------|
| hello | 17 | 2019-08-16 00:52:07.0000000 |
| world | 71 | 2019-08-16 00:52:08.0000000 |
| isn't, this neat? | -13 | 2019-08-16 00:52:09.0000000 |
Related
I've successfully been using jq for a while now, to take a JSON payload, select some of the columns, re-name the columns, and finally, create a JSON file. This is awesome bc I do not need a majority of the columns in the input dataset. Here is an example of one of those working commands:
curl -s https://c2.scryfall.com/file/scryfall-bulk/default-cards/default-cards-20220314210303.json
| jq '[.[] | {oracle_id: .oracle_id, scryfall_id: .id, rarity: .rarity, set_code: .set, latest_price: .prices.usd, scryfall_url: .scryfall_uri, art_crop_url: .image_uris.art_crop, is_digital: .digital, is_promo: .promo, is_variation: .variation}]' > Desktop/printings.json
However, what I really need is to have this data in CSV format. I have been manually working around this by feeding the output of the command above into a free web tool for converting to CSV. But I recently learned that jq can output CSV itself, so I would like to streamline this so I can just get CSV data from jq in the first place. I read the jq documentation, and reviewed several Stack Overflow threads to learn how this works. But none of the examples I've found for generating CSV data with jq involve selecting specific columns or re-naming those columns. So I've not been able to get this to work.
I tried this command below, where I am attempting to 1) read in the JSON file from the scryfall.com endpoint, then 2) map the keys as rows and columns to prep to convert to the CSV format, and 3) apply a filter selecting each of the 10 columns I need. (I could not figure out the column re-naming part, so I removed that part for now, for the sake of simplicity):
curl -s https://c2.scryfall.com/file/scryfall-bulk/default-cards/default-cards-20220314210303.json
| jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | .oracle_id | .id | .rarity | .set | .prices.usd | .scryfall_uri | .image_uris.art_crop | .digital | .promo | .variation | #csv' > Desktop/printings.csv
The result is this error:
jq: error (at <stdin>:67121): Cannot index array with string "oracle_id"
I'm not sure why "| .oracle_id" would be indexing anything. My intent is to filter the data. However, I think my struggle is an algorithmic one. Should I try to use pipes to sequence the different steps of selecting columns and generating the csv? Or should I combine them? If I need to separate the steps, what order do they need to come in? I understnad that the #csv filter at the end must take an array as input, but that's where I start to lose the plot.
Since the input JSON file is a freely-available, public dataset, you should be able to try this out to see if you get the same error output I showed above.
In general, you should try breaking out each "group" and testing separately, to see if it is mapping as you expect it to.
$cols maps out all the keys across all records while $rows are all the values of the records. You already have the rows and records you wanted so pass to #csv. Though keep in mind arrays passed to #csv must be all strings.
(map(keys) | add | unique) as $cols
| map(. as $row | $cols | map($row[.] | tostring)) as $rows
| $cols, $rows[]
| #csv
This however selects all mapped columns. If you only want a subset of them, just change the $cols variable to be what columns you want from the data. You might want to separate the value mapping from this since you have some nested values.
["oracle_id", "id", "rarity", "set", "price", "scryfall_uri", "image_uri", "digital", "promo", "variation"],
(.[] | [.oracle_id, .id, .rarity, .set, .prices.usd, .scryfall_uri, .image_uris.art_crop, .digital, .promo, .variation])
| #csv
jqplay
We are using Forge to import a STEP file into the modelspace of an output.DWG. Then a DLL combines modelspace geometry of several DWG files into several layout/paperspace of a single DWG. This sheet combination was working perfectly until just recently, when the combination process completely stopped happening.
Has something in Forge changed recently that we're not aware of? Updates/patches, or something like that which could have caused this issue?
This is an issue for a production application and is considered an outage at this point, and is very time-sensitive.
Edit: Here are some differences we noticed between the log files generated by this process. In this first section, the verbiage being written by AutoCAD has changed slightly during an extraction process:
[08/01/2019 17:15:35] End downloading https://.... 1556909 bytes have been unpacked to folder T:\Aces\Jobs\a43e5ca7faaa4db8b5374aaef71b36d3\cadlayouts.
[08/19/2019 17:25:53] End downloading file https://.... 1771363 bytes have been written to T:\Aces\Jobs\d12f3bed13b84d29b31226222e3cf3c9\cadlayouts.
In the log from 8/19, all lines logged in between:
Start AutoCAD Core Engine standard output dump.
And:
End AutoCAD Core Engine standard output dump.
Are being written twice, but this did not happen in the log file from August 1st or any of the logs before that date.
Edit 2:
Yesterday we used the .NET DirectoryInfo class to pull all directories into one list and all files into another and write them all to the log. The cadlayouts entity that should be recognized as a directory (because it's a zip that is extracted by Forge) is instead listed as a file. Our process runs a Directory.Exists() check before the work item merges the DWGs into the output, and this call returns false for the cadlayouts folder, bypassing our combination logic. How can the Forge zip extraction process be working correctly if the resulting entity on the file system is not considered a directory?
It sounds like you have an input argument that is a zip and you expect it to be unzipped into a folder. Please look row 4 in the table below. I suspect that this is what you are experiencing. There WAS a recent change here: we used to look at downloaded bits and unconditionally uncompressed if we found a zip header. (i.e. we acted identically for row 3 and row 4). We now only do this if you ask us to do it.
EDIT: The first column in the table is the value of the zip attribute of Activity's parameters while the second column is the pathInzip attribute of Workitem's arguments.
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| # | Activity | Workitem | Arg direction | Comments |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 1 | zip==true | pathInZip!=null | input | Zip is uncompressed to the folder specified in localname. Any path reference to this argument will expand to full path of pathInZip. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2 | zip==false | pathInZip!=null | input | Zip is uncompressed to the folder specified in localname. Any path reference to this argument will expand to full path of pathInZip. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 3 | zip==true | pathInZip==null | input | If zip is provided then it is uncompressed to the folder specified in localname. Any path reference to this argument will expand to full path of localName. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 4 | zip==false | pathInZip==null | input | If zip is provided then it is left compressed. Any variable referencing this argument will expand to full path of localName. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 5 | zip==true | pathInZip!=null | output | Workitem will be rejected. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 6 | zip==false | pathInZip!=null | output | Workitem will be rejected. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 7 | zip==true | pathInZip==null | output | Output(s) at localName will be zipped if localName is a folder. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 8 | zip==false | pathInZip==null | output | Output at localName will not be zipped. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
I want to export a cassandra DB to a csv file, but
cqlsh:marvel> SELECT * FROM personajes ;
name | skills
------------+--------
Iron Man | Tech
Spider Man | Lab
cqlsh:marvel> COPY personajes (name, skills) TO 'temp.csv';
get_num_processes() takes no keyword arguments
Tested in:
[cqlsh 5.0.1 | Cassandra 2.1.14 | CQL spec 3.2.1 | Native protocol v3]
[cqlsh 5.0.1 | Cassandra 3.0.5 | CQL spec 3.4.0 | Native protocol v4]
Thank you
Delete pylib/cqlshlib/copyutil.so and copyutil.c (if you have it - I didn't).
The exact path depends on your OS I guess. On Ubuntu 14.04 copyutil.so would be a symlink inside /usr/lib/pymodules/python2.7/cqlshlib.
Just delete or rename it and you should be good to go. Worked for me at least.
For reference: This is indeed a bug and the same bug as https://issues.apache.org/jira/browse/CASSANDRA-11574, which I've learned after opening https://issues.apache.org/jira/browse/CASSANDRA-11816. Turned out the fix-version in the first ticket was wrong for Cassandra 2.2
I Am trying to load a json data file into a variable directly in Robot Framework. Can anyone please elaborate with an e.g. giving the exact syntax as to how to do it?
Thanks in advance :)
One way would be to use the Get File keyword from the OperatingSystem library, and then use the built-in Evaluate keyword to convert it to a python object.
For example, consider a file named example.json with the following contents:
{
"firstname": "Inigo",
"lastname": "Montoya"
}
You can log the name with something like this:
*** Settings ***
| Library | OperatingSystem
*** Test Cases ***
| Example of how to load JSON
| | # read the raw data
| | ${json}= | Get file | example.json
| |
| | # convert the data to a python object
| | ${object}= | Evaluate | json.loads('''${json}''') | json
| |
| | # log the data
| | log | Hello, my name is ${object["firstname"]} ${object["lastname"]} | WARN
Of course, you could also write your own library in python to create a keyword that does the same thing.
There is a library available for this: HttpLibrary.HTTP
${json}= | Get file | example.json
${port} | HttpLibrary.HTTP.Get Json Value | ${json} | /port
log | ${port}
API Document is available here: http://peritus.github.io/robotframework-httplibrary/HttpLibrary.html
A common use is passing the json data to another library like Http Library Requests. You could do:
*** Settings ***
Library OperatingSystem
Library RequestsLibrary
*** Test Cases ****
Create User
#...
${file_data}=
... Get Binary File ${RESOURCES}${/}normal_user.json
Post Request example_session /user data=${file_data}
#...
No direct python involved and no intermediary json object.
Thanks Vinay .. that helped now we can retrieve data from json file in robot framework as well
*** Settings ***
Library HttpLibrary.HTTP
Library OperatingSystem
*** Test Cases ***
Login_to_SalesForce_Json
${jsonfile} Get File c:/pathtojason/Data/testsuite.json
${username} Get Json Value ${jsonfile} /test_case1/username
log ${username}
Below is the json file structure
{
"test_case1":
{
"username":"User1",
"password":"Pass1"
}
,
"test_case2":
{
"username1":"User2",
"password1":"Pass2"
}
}
Prerequiste is:pip install --trusted-host pypi.python.org robotframework-httplibrary
I had similar issue and this work fine with me:
${json} Get Binary File ${json_path}nameOfJsonFile.json
It works for me on API testing, to read .json and POST, like here
*** Settings ***
Library Collections
Library ExtendedRequestsLibrary
Library OperatingSystem
*** Variables ***
${uri} https://blabla.com/service/
${json_path} C:/home/user/project/src/json/
*** Test Cases ***
Name of Robot Test Case
Create Session alias ${uri}
&{headers} Create Dictionary Content-Type=application/json; charset=utf-8
${json} Get Binary File ${json_path}nameOfJsonFile.json
${resp} Post Request alias data=${shiftB} headers=${headers}
Should Be Equal As Strings ${resp.status_code} 200
There are also cases when you will need to transform read binary file (in my case ${json} to a dictionary but first try this simple solution.
For example CSV and JSON are human and machine readable text formats.
Now I am looking for something similar even more graphical for table data representation.
Instead of:
1,"machines",14.91
3,"mammals",1.92
50,"fruit",4.239
789,"funghi",29.3
which is CSV style or
[
[1,"machines",14.91],
[3,"mammals",1.92],
[50,"fruit",4.239],
[789,"funghi",29.3]
]
which is JSON style, and I am not going to give an XML example, something similar like this is what I have in mind:
1 | "machines"| 14.91
3 | "mammals" | 1.92
50 | "fruit" | 4.239
789 | "funghi" | 29.3
There should be reader and writer libraries for it for some languages and it should somehow be a standard. Of course I could roll my own but if there is also a standard I'd go with that.
I have seen similar things as part of wiki or markup languages, but it should serve as a human easily editable data definition format and be read and also written by software libraries.
That's not exactly what markup and wiki languages are for. What I am looking for belongs more to the csv,json and xml family.
I would checkout textile. It has a table syntax almost exactly like what you described.
For example, the table in your example would be constructed like this:
| 1 | machines | 14.91 |
| 3 | mammals | 1.92 |
| 50 | fruit | 4.239 |
| 789 | funghi | 29.3 |
An alternative (albeit not optimized for tabular data), is YAML, which is nice for JSON-ish type data.
Alternatively you could also look at the CSV editor's i.e.
CsvEd
CsvEasy
ReCsvEditor
There whole purpose is to display CSV and update data in a more readable Format. The ReCsvEditor will display both Xml and Csv files in a a similar format.
Google CsvEditor, you will find plenty