For example CSV and JSON are human and machine readable text formats.
Now I am looking for something similar even more graphical for table data representation.
Instead of:
1,"machines",14.91
3,"mammals",1.92
50,"fruit",4.239
789,"funghi",29.3
which is CSV style or
[
[1,"machines",14.91],
[3,"mammals",1.92],
[50,"fruit",4.239],
[789,"funghi",29.3]
]
which is JSON style, and I am not going to give an XML example, something similar like this is what I have in mind:
1 | "machines"| 14.91
3 | "mammals" | 1.92
50 | "fruit" | 4.239
789 | "funghi" | 29.3
There should be reader and writer libraries for it for some languages and it should somehow be a standard. Of course I could roll my own but if there is also a standard I'd go with that.
I have seen similar things as part of wiki or markup languages, but it should serve as a human easily editable data definition format and be read and also written by software libraries.
That's not exactly what markup and wiki languages are for. What I am looking for belongs more to the csv,json and xml family.
I would checkout textile. It has a table syntax almost exactly like what you described.
For example, the table in your example would be constructed like this:
| 1 | machines | 14.91 |
| 3 | mammals | 1.92 |
| 50 | fruit | 4.239 |
| 789 | funghi | 29.3 |
An alternative (albeit not optimized for tabular data), is YAML, which is nice for JSON-ish type data.
Alternatively you could also look at the CSV editor's i.e.
CsvEd
CsvEasy
ReCsvEditor
There whole purpose is to display CSV and update data in a more readable Format. The ReCsvEditor will display both Xml and Csv files in a a similar format.
Google CsvEditor, you will find plenty
Related
I have a csv file with the following single line:
Some,Test,"is"thisvalid,or,not
according to cvslint.io it's not valid csv:
However according to https://www.toolkitbay.com/tkb/tool/csv-validator it is valid csv. Which site is lying?
Whether it is "valid" or not depends on the definition you, and the websites you found, are using. If you asked about "well-formed XML", everyone would agree that should be based on the W3C standard; or "valid HTML" would now probably refer to the WHATWG Living Standard. "Valid CSV" has no such universal definition - although there are standards for CSV, they've been written after years of use, in the rather optimistic hope that existing implementations will be amended to follow them.
So neither tool is "lying", they just evidently disagree on what "valid" means.
A far more useful question than if CSV is "valid" is whether it is interpreted as you want by whatever tool you try to process it with. From a practical point of view, it's likely that the unusual positioning of quote marks might be interpreted differently by different tools, so is probably best avoided if interoperability is relevant to your use case.
For CSV format the reference is this https://datatracker.ietf.org/doc/html/rfc4180
And you have:
Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes.
If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote
Then your CSV is not valid.
If your columns are these
+------+------+---------------+----+-----+
| 1 | 2 | 3 | 4 | 5 |
+------+------+---------------+----+-----+
| Some | Test | "is"thisvalid | or | not |
+------+------+---------------+----+-----+
then valid version is this
Some,Test,"""is""thisvalid",or,not
And it's valid also for https://csvlint.io/
I've successfully been using jq for a while now, to take a JSON payload, select some of the columns, re-name the columns, and finally, create a JSON file. This is awesome bc I do not need a majority of the columns in the input dataset. Here is an example of one of those working commands:
curl -s https://c2.scryfall.com/file/scryfall-bulk/default-cards/default-cards-20220314210303.json
| jq '[.[] | {oracle_id: .oracle_id, scryfall_id: .id, rarity: .rarity, set_code: .set, latest_price: .prices.usd, scryfall_url: .scryfall_uri, art_crop_url: .image_uris.art_crop, is_digital: .digital, is_promo: .promo, is_variation: .variation}]' > Desktop/printings.json
However, what I really need is to have this data in CSV format. I have been manually working around this by feeding the output of the command above into a free web tool for converting to CSV. But I recently learned that jq can output CSV itself, so I would like to streamline this so I can just get CSV data from jq in the first place. I read the jq documentation, and reviewed several Stack Overflow threads to learn how this works. But none of the examples I've found for generating CSV data with jq involve selecting specific columns or re-naming those columns. So I've not been able to get this to work.
I tried this command below, where I am attempting to 1) read in the JSON file from the scryfall.com endpoint, then 2) map the keys as rows and columns to prep to convert to the CSV format, and 3) apply a filter selecting each of the 10 columns I need. (I could not figure out the column re-naming part, so I removed that part for now, for the sake of simplicity):
curl -s https://c2.scryfall.com/file/scryfall-bulk/default-cards/default-cards-20220314210303.json
| jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | .oracle_id | .id | .rarity | .set | .prices.usd | .scryfall_uri | .image_uris.art_crop | .digital | .promo | .variation | #csv' > Desktop/printings.csv
The result is this error:
jq: error (at <stdin>:67121): Cannot index array with string "oracle_id"
I'm not sure why "| .oracle_id" would be indexing anything. My intent is to filter the data. However, I think my struggle is an algorithmic one. Should I try to use pipes to sequence the different steps of selecting columns and generating the csv? Or should I combine them? If I need to separate the steps, what order do they need to come in? I understnad that the #csv filter at the end must take an array as input, but that's where I start to lose the plot.
Since the input JSON file is a freely-available, public dataset, you should be able to try this out to see if you get the same error output I showed above.
In general, you should try breaking out each "group" and testing separately, to see if it is mapping as you expect it to.
$cols maps out all the keys across all records while $rows are all the values of the records. You already have the rows and records you wanted so pass to #csv. Though keep in mind arrays passed to #csv must be all strings.
(map(keys) | add | unique) as $cols
| map(. as $row | $cols | map($row[.] | tostring)) as $rows
| $cols, $rows[]
| #csv
This however selects all mapped columns. If you only want a subset of them, just change the $cols variable to be what columns you want from the data. You might want to separate the value mapping from this since you have some nested values.
["oracle_id", "id", "rarity", "set", "price", "scryfall_uri", "image_uri", "digital", "promo", "variation"],
(.[] | [.oracle_id, .id, .rarity, .set, .prices.usd, .scryfall_uri, .image_uris.art_crop, .digital, .promo, .variation])
| #csv
jqplay
Unable to insert data from object array or csv file into kusto table
My goal is to build a pipeline in Azure DevOps which reads data using PowerShell and writes the data into Kusto Table.
I was able to write the data which I have read from PowerShell to object Array or csv file but I am unable to figure out the ways in which this data can be inserted into Kusto table.
Could any one suggest the best way to write the data into kusto
one option would be to write your CSV payload to blob storage, then ingest that blob into your target table, by:
using a "queued ingestion" client in one of the client libraries: https://learn.microsoft.com/en-us/azure/kusto/api/
note that the .NET ingestion client library also provides you with methods to IngestFromStream or IngestFromDataReader, which handle writing the data to intermediate blob storage so that you don't have to
or by
issuing an .ingest command: https://learn.microsoft.com/en-us/azure/kusto/management/data-ingestion/ingest-from-storage. though using "direction ingestion" is less recommended for Production volumes
another option (not recommended for Production volume), would be using the .ingest inline (AKA "ingest push") option: https://learn.microsoft.com/en-us/azure/kusto/management/data-ingestion/ingest-inline
for example:
.create table sample_table (a:string, b:int, c:datetime)
.ingest inline into table sample_table <|
hello,17,2019-08-16 00:52:07
world,71,2019-08-16 00:52:08
"isn't, this neat?",-13,2019-08-16 00:52:09
which will append the above records to the table:
| a | b | c |
|-------------------|------|-----------------------------|
| hello | 17 | 2019-08-16 00:52:07.0000000 |
| world | 71 | 2019-08-16 00:52:08.0000000 |
| isn't, this neat? | -13 | 2019-08-16 00:52:09.0000000 |
We are using Forge to import a STEP file into the modelspace of an output.DWG. Then a DLL combines modelspace geometry of several DWG files into several layout/paperspace of a single DWG. This sheet combination was working perfectly until just recently, when the combination process completely stopped happening.
Has something in Forge changed recently that we're not aware of? Updates/patches, or something like that which could have caused this issue?
This is an issue for a production application and is considered an outage at this point, and is very time-sensitive.
Edit: Here are some differences we noticed between the log files generated by this process. In this first section, the verbiage being written by AutoCAD has changed slightly during an extraction process:
[08/01/2019 17:15:35] End downloading https://.... 1556909 bytes have been unpacked to folder T:\Aces\Jobs\a43e5ca7faaa4db8b5374aaef71b36d3\cadlayouts.
[08/19/2019 17:25:53] End downloading file https://.... 1771363 bytes have been written to T:\Aces\Jobs\d12f3bed13b84d29b31226222e3cf3c9\cadlayouts.
In the log from 8/19, all lines logged in between:
Start AutoCAD Core Engine standard output dump.
And:
End AutoCAD Core Engine standard output dump.
Are being written twice, but this did not happen in the log file from August 1st or any of the logs before that date.
Edit 2:
Yesterday we used the .NET DirectoryInfo class to pull all directories into one list and all files into another and write them all to the log. The cadlayouts entity that should be recognized as a directory (because it's a zip that is extracted by Forge) is instead listed as a file. Our process runs a Directory.Exists() check before the work item merges the DWGs into the output, and this call returns false for the cadlayouts folder, bypassing our combination logic. How can the Forge zip extraction process be working correctly if the resulting entity on the file system is not considered a directory?
It sounds like you have an input argument that is a zip and you expect it to be unzipped into a folder. Please look row 4 in the table below. I suspect that this is what you are experiencing. There WAS a recent change here: we used to look at downloaded bits and unconditionally uncompressed if we found a zip header. (i.e. we acted identically for row 3 and row 4). We now only do this if you ask us to do it.
EDIT: The first column in the table is the value of the zip attribute of Activity's parameters while the second column is the pathInzip attribute of Workitem's arguments.
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| # | Activity | Workitem | Arg direction | Comments |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 1 | zip==true | pathInZip!=null | input | Zip is uncompressed to the folder specified in localname. Any path reference to this argument will expand to full path of pathInZip. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2 | zip==false | pathInZip!=null | input | Zip is uncompressed to the folder specified in localname. Any path reference to this argument will expand to full path of pathInZip. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 3 | zip==true | pathInZip==null | input | If zip is provided then it is uncompressed to the folder specified in localname. Any path reference to this argument will expand to full path of localName. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 4 | zip==false | pathInZip==null | input | If zip is provided then it is left compressed. Any variable referencing this argument will expand to full path of localName. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 5 | zip==true | pathInZip!=null | output | Workitem will be rejected. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 6 | zip==false | pathInZip!=null | output | Workitem will be rejected. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 7 | zip==true | pathInZip==null | output | Output(s) at localName will be zipped if localName is a folder. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 8 | zip==false | pathInZip==null | output | Output at localName will not be zipped. |
+---+------------+-----------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
I noticed that depending on the implementation, some JSON libraries quote / characters, others don't.
Example 1: Lua
local cjson = require 'cjson'
print(cjson.encode({ x = "/" }))
--> {"x":"\/"}
Example 2: JavaScript
console.log(JSON.stringify({ x: "/" }))
--> '{"x":"/"}'
I wonder if the quoting of Lua's cjson libray is a bug or a valid feature. If it is not, I'm concerned about base64 encoded strings that are sent over the network and should be processed by any language. I'm concerned about possibly unintended side-effects of Lua cjson when it changes strings after first decoding the JSON string and than encoding it again, for example:
local x = '{"x":"/"}'
print(x)
--> {"x":"/"}
print(cjson.encode(cjson.decode(x)))
--> {"x":"\/"}
I wonder if this is allowed. Is it still the same JSON data? I would have expected that the actual string contents should not be changed by applying a decode followed by an encode operation.
Is it allowed in JSON to quote a '/', or does it change the payload in a non standard conformant way?
From what I tested, assuming that "/" == "\/" holds is not portable over different languages. In a small sample of languages, I found mixed results. Some accept it, some don't, some accept it but issue warnings (so it is maybe not portable). Here is an overview:
+------------+-------------+----------------------------------+
| Language | "/" == "\/" | Notes |
+------------+-------------+----------------------------------+
| Lua | true | - |
| JavaScript | true | - |
| C++ | true | warning: unknown escape sequence |
| Python | false | - |
| Ruby | true | - |
+------------+-------------+----------------------------------+
The spec defines a string as
So the sequence \/ is clearly allowed. But it is not necessary, since / also falls into the "Any Unicode character except " or \ or control character" range.
The "warning: unknown escape sequence" is not correct in this case.
If it is not, I'm concerned about base64 encoded strings that are sent over the network and should be processed by any language.
I'm not sure I understand. Base64 and JSON have nothing to do with each other.
Going by the ECMA-404 spec, it should be allowed:
\/ represents the solidus character (U+002F).
The following four cases all produce the same result:
"\u002F"
"\u002f"
"\/"
"/"