Push JSON file with special chars unicode to bigquery via command line - json

Im trying to push a json file to BQ via bqcmd which contains special chars such as ü but tis failing. Ive tried differnt encoding etc. nothing workign except removing the chars
{
"productName": "soft drüum",
"Category": "xxxxxxx",
"subCategory": "Sun & Tan",
"subSubCategory": "Sun",
"productType": "Standard",
"brandName": "soft drüum",
"quantity": 1
}
Json File was successful until introduction of special chars
BQ command:
bq load --autodetect=true --time_partitioning_field date
--source_format NEWLINE_DELIMITED_JSON Data.products D:\XXXX\xx\products.json


I would check to make sure that you have your json saved as new line delimited. In your example, the trailing comma is not valid in New Line Delimited JSON.
I was able to then upload these special characters via this command:
bq load --project_id=${project_id} --source_format NEWLINE_DELIMITED_JSON --autodetect ${database}.${table} ${nldjson_filename}

Related

Powershell - Cannot parse JSON - Get NotSpecified error

Code used to try parsing file
$myJson = Get-Content cleantest2.json -Raw | ConvertFrom-Json
Get below error when trying to parse JSON file.
$myJson = Get-Content cleantest2.json -Raw | ConvertFrom-Json
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand
EDIT: As noted, the above error was truncated because it was too long. Here is the full error.
ConvertFrom-Json : Invalid object passed in, ':' or '}' expected. (2): {"id":0,"resourcePackage
s":[{"resourcePackage":{"name":"SharedResources","type":2,"i
tems":[{"type":202,"path":"BaseThemes/CY18SU07.json","name"
:"CY18SU07"}],"disabled":false}}],"sections":[{"id":0,"name"
:"ReportSection","displayName":"Page 1","filters":"[]","ord
inal":0,"visualContainers":[{"x":10,"y":0,"z":1,"width":280,
"height":280,"config":"{\"name\":\"5a0b8ef6c97162087ff7\",\
"layouts\":[{\"id\":0,\"position\":{\"x\":10,\"y\":0,\"width
\":280,\"height\":280,\"z\":1}}],\"singleVisual\":{\"visual
Type\":\"tableEx\",\"projections\":{\"Values\":[{\"queryRef\
":\"servers_and_applications.App Name\"},{\"queryRef\":\"se
rvers_and_applications.Cust_Nme\"},{\"queryRef\":\"servers_a
nd_applications.MU_ID\"}]},\"prototypeQuery\":{\"Version\":
2,\"From\":[{\"Name\":\"s\",\"Entity\":\"servers_and_applica
tions\"}],\"Select\":[{\"Column\":{\"Expression\":{\"Source
Ref\":{\"Source\":\"s\"}},\"Property\":\"App
Name\"},\"Name\":\"servers_and_applications.App Name\"},{\"
Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},
\"Property\":\"Cust_Nme\"},\"Name\":\"servers_and_applicati
ons.Cust_Nme\"},{\"Column\":{\"Expression\":{\"SourceRef\":{
\"Source\":\"s\"}},\"Property\":\"MU_ID\"},\"Name\":\"serve
rs_and_applications.MU_ID\"}]},\"drillFilterOtherVisuals\":t
rue}}","filters":"[]","query":"{\"Commands\":[{\"SemanticQu
eryDataShapeCommand\":{\"Query\":{\"Version\":2,\"From\":[{\
"Name\":\"s\",\"Entity\":\"servers_and_applications\"}],\"S
elect\":[{\"Column\":{\"Expression\":{\"SourceRef\":{\"Sourc
e\":\"s\"}},\"Property\":\"App
Name\"},\"Name\":\"servers_and_applications.App Name\"},{\"
Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},
\"Property\":\"Cust_Nme\"},\"Name\":\"servers_and_applicati
ons.Cust_Nme\"},{\"Column\":{\"Expression\":{\"SourceRef\":{
\"Source\":\"s\"}},\"Property\":\"MU_ID\"},\"Name\":\"serve
rs_and_applications.MU_ID\"}]},\"Binding\":{\"Primary\":{\"G
roupings\":[{\"Projections\":[0,1,2],\"Subtotal\":1}]},\"Da
taReduction\":{\"DataVolume\":3,\"Primary\":{\"Window\":{\"C
ount\":500}}},\"Version\":1}}}]}","dataTransforms":"{\"proj
ectionOrdering\":{\"Values\":[0,1,2]},\"queryMetadata\":{\"S
elect\":[{\"Restatement\":\"App
Name\",\"Name\":\"servers_and_applications.App Name\",\"Typ
e\":2048},{\"Restatement\":\"Cust_Nme\",\"Name\":\"servers_a
nd_applications.Cust_Nme\",\"Type\":2048},{\"Restatement\":
\"MU_ID\",\"Name\":\"servers_and_applications.MU_ID\",\"Type
\":2048}]},\"visualElements\":[{\"DataRoles\":[{\"Name\":\"
Values\",\"Projection\":0,\"isActive\":false},{\"Name\":\"Va
lues\",\"Projection\":1,\"isActive\":false},{\"Name\":\"Val
ues\",\"Projection\":2,\"isActive\":false}]}],\"selects\":[{
\"displayName\":\"App
Name\",\"queryName\":\"servers_and_applications.App Name\",
\"roles\":{\"Values\":true},\"type\":{\"category\":null,\"un
derlyingType\":1},\"expr\":{\"Column\":{\"Expression\":{\"S
ourceRef\":{\"Entity\":\"servers_and_applications\"}},\"Prop
erty\":\"App Name\"}}},{\"displayName\":\"Cust_Nme\",\"quer
yName\":\"servers_and_applications.Cust_Nme\",\"roles\":{\"V
alues\":true},\"type\":{\"category\":null,\"underlyingType\
":1},\"expr\":{\"Column\":{\"Expression\":{\"SourceRef\":{\"
Entity\":\"servers_and_applications\"}},\"Property\":\"Cust
_Nme\"}}},{\"displayName\":\"MU_ID\",\"queryName\":\"servers
_and_applications.MU_ID\",\"roles\":{\"Values\":true},\"typ
e\":{\"category\":null,\"underlyingType\":1},\"expr\":{\"Col
umn\":{\"Expression\":{\"SourceRef\":{\"Entity\":\"servers_
and_applications\"}},\"Property\":\"MU_ID\"}}}]}"}],"config"
:"{}","displayOption":1,"width":1280,"height":720}],"config
":"{\"version\":\"3.19\",\"themeCollection\":{\"baseTheme\":
{\"name\":\"CY18SU07\",\"version\":\"3.22\",\"type\":2}},\"
activeSectionIndex\":0,\"defaultDrillFilterOtherVisuals\":tr
ue,\"settings\":{\"useStylableVisualContainerHeader\":true,
\"exportDataMode\":1}}","layoutOptimization":0}
At line:1 char:46
+ $myJson = Get-Content cleantest2.json -Raw | ConvertFrom-Json
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand
Below is file that was trying to get parsed.
{"id":0,"resourcePackages":[{"resourcePackage":{"name":"SharedResources","type":2,"items":[{"type":202,"path":"BaseThemes/CY18SU07.json","name":"CY18SU07"}],"disabled":false}}],"sections":[{"id":0,"name":"ReportSection","displayName":"Page 1","filters":"[]","ordinal":0,"visualContainers":[{"x":10,"y":0,"z":1,"width":280,"height":280,"config":"{\"name\":\"5a0b8ef6c97162087ff7\",\"layouts\":[{\"id\":0,\"position\":{\"x\":10,\"y\":0,\"width\":280,\"height\":280,\"z\":1}}],\"singleVisual\":{\"visualType\":\"tableEx\",\"projections\":{\"Values\":[{\"queryRef\":\"servers_and_applications.App Name\"},{\"queryRef\":\"servers_and_applications.Cust_Nme\"},{\"queryRef\":\"servers_and_applications.MU_ID\"}]},\"prototypeQuery\":{\"Version\":2,\"From\":[{\"Name\":\"s\",\"Entity\":\"servers_and_applications\"}],\"Select\":[{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"App Name\"},\"Name\":\"servers_and_applications.App Name\"},{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"Cust_Nme\"},\"Name\":\"servers_and_applications.Cust_Nme\"},{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"MU_ID\"},\"Name\":\"servers_and_applications.MU_ID\"}]},\"drillFilterOtherVisuals\":true}}","filters":"[]","query":"{\"Commands\":[{\"SemanticQueryDataShapeCommand\":{\"Query\":{\"Version\":2,\"From\":[{\"Name\":\"s\",\"Entity\":\"servers_and_applications\"}],\"Select\":[{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"App Name\"},\"Name\":\"servers_and_applications.App Name\"},{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"Cust_Nme\"},\"Name\":\"servers_and_applications.Cust_Nme\"},{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"MU_ID\"},\"Name\":\"servers_and_applications.MU_ID\"}]},\"Binding\":{\"Primary\":{\"Groupings\":[{\"Projections\":[0,1,2],\"Subtotal\":1}]},\"DataReduction\":{\"DataVolume\":3,\"Primary\":{\"Window\":{\"Count\":500}}},\"Version\":1}}}]}","dataTransforms":"{\"projectionOrdering\":{\"Values\":[0,1,2]},\"queryMetadata\":{\"Select\":[{\"Restatement\":\"App Name\",\"Name\":\"servers_and_applications.App Name\",\"Type\":2048},{\"Restatement\":\"Cust_Nme\",\"Name\":\"servers_and_applications.Cust_Nme\",\"Type\":2048},{\"Restatement\":\"MU_ID\",\"Name\":\"servers_and_applications.MU_ID\",\"Type\":2048}]},\"visualElements\":[{\"DataRoles\":[{\"Name\":\"Values\",\"Projection\":0,\"isActive\":false},{\"Name\":\"Values\",\"Projection\":1,\"isActive\":false},{\"Name\":\"Values\",\"Projection\":2,\"isActive\":false}]}],\"selects\":[{\"displayName\":\"App Name\",\"queryName\":\"servers_and_applications.App Name\",\"roles\":{\"Values\":true},\"type\":{\"category\":null,\"underlyingType\":1},\"expr\":{\"Column\":{\"Expression\":{\"SourceRef\":{\"Entity\":\"servers_and_applications\"}},\"Property\":\"App Name\"}}},{\"displayName\":\"Cust_Nme\",\"queryName\":\"servers_and_applications.Cust_Nme\",\"roles\":{\"Values\":true},\"type\":{\"category\":null,\"underlyingType\":1},\"expr\":{\"Column\":{\"Expression\":{\"SourceRef\":{\"Entity\":\"servers_and_applications\"}},\"Property\":\"Cust_Nme\"}}},{\"displayName\":\"MU_ID\",\"queryName\":\"servers_and_applications.MU_ID\",\"roles\":{\"Values\":true},\"type\":{\"category\":null,\"underlyingType\":1},\"expr\":{\"Column\":{\"Expression\":{\"SourceRef\":{\"Entity\":\"servers_and_applications\"}},\"Property\":\"MU_ID\"}}}]}"}],"config":"{}","displayOption":1,"width":1280,"height":720}],"config":"{\"version\":\"3.19\",\"themeCollection\":{\"baseTheme\":{\"name\":\"CY18SU07\",\"version\":\"3.22\",\"type\":2}},\"activeSectionIndex\":0,\"defaultDrillFilterOtherVisuals\":true,\"settings\":{\"useStylableVisualContainerHeader\":true,\"exportDataMode\":1}}","layoutOptimization":0}
Reformatted:
{
"id": 0,
"resourcePackages": [
{
"resourcePackage": {
"name": "SharedResources",
"type": 2,
"items": [
{
"type": 202,
"path": "BaseThemes/CY18SU07.json",
"name": "CY18SU07"
}
],
"disabled": false
}
}
],
"sections": [
{
"id": 0,
"name": "ReportSection",
"displayName": "Page 1",
"filters": "[]",
"ordinal": 0,
"visualContainers": [
{
"x": 10,
"y": 0,
"z": 1,
"width": 280,
"height": 280,
"config": "{\"name\":\"5a0b8ef6c97162087ff7\",\"layouts\":[{\"id\":0,\"position\":{\"x\":10,\"y\":0,\"width\":280,\"height\":280,\"z\":1}}],\"singleVisual\":{\"visualType\":\"tableEx\",\"projections\":{\"Values\":[{\"queryRef\":\"servers_and_applications.App Name\"},{\"queryRef\":\"servers_and_applications.Cust_Nme\"},{\"queryRef\":\"servers_and_applications.MU_ID\"}]},\"prototypeQuery\":{\"Version\":2,\"From\":[{\"Name\":\"s\",\"Entity\":\"servers_and_applications\"}],\"Select\":[{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"App Name\"},\"Name\":\"servers_and_applications.App Name\"},{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"Cust_Nme\"},\"Name\":\"servers_and_applications.Cust_Nme\"},{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"MU_ID\"},\"Name\":\"servers_and_applications.MU_ID\"}]},\"drillFilterOtherVisuals\":true}}",
"filters": "[]",
"query": "{\"Commands\":[{\"SemanticQueryDataShapeCommand\":{\"Query\":{\"Version\":2,\"From\":[{\"Name\":\"s\",\"Entity\":\"servers_and_applications\"}],\"Select\":[{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"App Name\"},\"Name\":\"servers_and_applications.App Name\"},{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"Cust_Nme\"},\"Name\":\"servers_and_applications.Cust_Nme\"},{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"s\"}},\"Property\":\"MU_ID\"},\"Name\":\"servers_and_applications.MU_ID\"}]},\"Binding\":{\"Primary\":{\"Groupings\":[{\"Projections\":[0,1,2],\"Subtotal\":1}]},\"DataReduction\":{\"DataVolume\":3,\"Primary\":{\"Window\":{\"Count\":500}}},\"Version\":1}}}]}",
"dataTransforms": "{\"projectionOrdering\":{\"Values\":[0,1,2]},\"queryMetadata\":{\"Select\":[{\"Restatement\":\"App Name\",\"Name\":\"servers_and_applications.App Name\",\"Type\":2048},{\"Restatement\":\"Cust_Nme\",\"Name\":\"servers_and_applications.Cust_Nme\",\"Type\":2048},{\"Restatement\":\"MU_ID\",\"Name\":\"servers_and_applications.MU_ID\",\"Type\":2048}]},\"visualElements\":[{\"DataRoles\":[{\"Name\":\"Values\",\"Projection\":0,\"isActive\":false},{\"Name\":\"Values\",\"Projection\":1,\"isActive\":false},{\"Name\":\"Values\",\"Projection\":2,\"isActive\":false}]}],\"selects\":[{\"displayName\":\"App Name\",\"queryName\":\"servers_and_applications.App Name\",\"roles\":{\"Values\":true},\"type\":{\"category\":null,\"underlyingType\":1},\"expr\":{\"Column\":{\"Expression\":{\"SourceRef\":{\"Entity\":\"servers_and_applications\"}},\"Property\":\"App Name\"}}},{\"displayName\":\"Cust_Nme\",\"queryName\":\"servers_and_applications.Cust_Nme\",\"roles\":{\"Values\":true},\"type\":{\"category\":null,\"underlyingType\":1},\"expr\":{\"Column\":{\"Expression\":{\"SourceRef\":{\"Entity\":\"servers_and_applications\"}},\"Property\":\"Cust_Nme\"}}},{\"displayName\":\"MU_ID\",\"queryName\":\"servers_and_applications.MU_ID\",\"roles\":{\"Values\":true},\"type\":{\"category\":null,\"underlyingType\":1},\"expr\":{\"Column\":{\"Expression\":{\"SourceRef\":{\"Entity\":\"servers_and_applications\"}},\"Property\":\"MU_ID\"}}}]}"
}
],
"config": "{}",
"displayOption": 1,
"width": 1280,
"height": 720
}
],
"config": "{\"version\":\"3.19\",\"themeCollection\":{\"baseTheme\":{\"name\":\"CY18SU07\",\"version\":\"3.22\",\"type\":2}},\"activeSectionIndex\":0,\"defaultDrillFilterOtherVisuals\":true,\"settings\":{\"useStylableVisualContainerHeader\":true,\"exportDataMode\":1}}",
"layoutOptimization": 0
}
Items attempted:
Used online formator on text and then re-ran command on resulting string. - Worked
Tried to import raw string and export as JSON using ConverTo-Json method but didn't work correctly.
Tried parsing using Python but had similar issue.

I was able to fix this by changing the encoding to unicode. See below sample execution.
$myJson = Get-Content cleantest2.json -Encoding unicode | ConvertFrom-Json

Your own answer implies that your input file, cleantest2.json, was malformed:
The fact that -Encoding Unicode solved your problem implies that the file is a UTF-16LE-encoded file that lacks a BOM, which means that it provides no indication to readers of the file what its encoding is, and causes Get-Content to (justifiably) misinterpret the file.
Consider correcting the problem at the source: make the creator of file cleantest2.json create it with the appropriate BOM, the way that Set-Content -Encoding Unicode does, for instance.
As for troubleshooting your symptom:
If a string's display output is seemingly correct, but processing of the string doesn't work as expected, the cause is usually the presence of hidden control characters.
Specifically, misreading a BOM-less UTF16-LE ("Unicode") file as either ANSI-encoded (which Windows PowerShell does by default) or UTF-8 (as PowerShell (Core) 7+ does by default) results in all ASCII-range characters being followed by a NUL character (code point 0x0)each, and these NUL characters are invisible when you print the string.
For testing, you can create a malformed UTF-16LE file as follows:
# Write 'foo' with UTF-16LE encoding to file 'cleantest2.json' *without a BOM*:
[IO.File]::WriteAllText("$PWD/cleantest2.json", "f`0o`0`o`0")
A simple test for the presence of at least one NUL character (expressed as "`0" in PowerShell) is:
(Get-Content cleantest2.json -Raw) -match "`0" # "`0" creates a NUL
Piping to Format-Hex can also help in this case:[1]
# Look for `00` values.
Get-Content cleantest2.json -Raw | Format-Hex
The Debug-String helper function, available from this MIT-licensed Gist offers a direct visualization:
# Download and define the Debug-String function.
# NOTE:
# I can personally assure you that doing this is safe, but you
# you should always check the source code first.
irm https://gist.github.com/mklement0/7f2f1e13ac9c2afaf0a0906d08b392d1/raw/Debug-String.ps1 | iex
Get-Content cleantest2.json -Raw | Debug-String
Output:
A note re spotting the problem with the help of text editors:
This may not help, because many editors correctly detect UTF-16LE files even without a BOM (e.g., Notepad, Visual Studio Code), and in their status bar report just "UTF-16 LE" whether or not a BOM is present.
However, if an editor offers a hex view (of the bytes that make up the file), you'll be able to tell: If an UTF-16LE file does have a BOM, the first two bytes are FF FE (for UTF-8 files that have a BOM, the first three bytes are EF BB BF).
Visual Studio Code has a Hex Editor extension
js2010 reports that Emacs has a hexl-mode.
[1] Use of Format-Hex to discover (non-ASCII-range) Unicode characters isn't an option in Windows PowerShell, because it represents any non-ASCII chars. as 3f, i.e. literal ?. In PowerShell (Core) it shows the ASCII rendering of the individual bytes that make up the UTF-8 encoding of non-ASCII characters, which can be confusing.

Adding Dollar Sign($) in a VScode Code snippet

I was writing my custom code snippets for verilog files in VScode; VSCode uses JSON files for it.
I observed that $ before finish and dumpvars statements doesn't get printed when I use the snippet since it is a built-in keyword for including variables in strings in json file format,
I have tried adding \ before $ but that didn't work.
Is there any way I can insert $ in my snippets?
This is the relevant code block I am using:
"dump statements": {
"prefix": "dump",
"body": [
"initial begin",
" dumpfile(\"${1:filename}\");",
" \$dumpvars();",
"end"
],
"description": "Prints the dump file and variables statements."
},

Use a double backslash to escape it, like this \\$
"dump statements": {
"prefix": "dump",
"body": [
"initial begin",
" dumpfile(\"${1:filename}\");",
" \\$dumpvars();",
"end"
],
"description": "Prints the dump file and variables statements."
},

Invalid JSON while submitting spark submit job via NiFi

I am trying to submit a spark job where I am setting a date argument in conf property and I am running it through a script in NiFi. However, when I am running the script I am facing an error.
Spark Submit Code in the script:
aws emr add-steps --cluster-id "$1" --steps '[{"Args":["spark-submit","--deploy-mode","cluster","--jars","s3://tvsc-lumiq-edl/jars/ojdbc7.jar","--executor-memory","10g","--driver-memory","10g","--conf","spark.hadoop.yarn.timeline-service.enabled=false","--conf","currDate='\"$5\"'","--class",'\"$2\"','\"$3\"','\"$4\"'],"Type":"CUSTOM_JAR","ActionOnFailure":"CONTINUE","Jar":"command-runner.jar","Properties":"","Name":"Spark application"}]' --region "$6"
and after I run it, I get the below error:
ExecuteStreamCommand[id=5b08df5a-1f24-3958-30ca-2e27a6c4becf] Transferring flow file StandardFlowFileRecord[uuid=00f844ee-dbea-42a3-aba3-0edcabfc50a2,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1607082757752-507103, container=default, section=223], offset=29, length=-1],offset=0,name=6414901712887990,size=0] to nonzero status. Executable command /bin/bash ended in an error:
Error parsing parameter '--steps': Invalid JSON:
[{"Args":["spark-submit","--deploy-mode","cluster","--jars","s3://tvsc-lumiq-edl/jars/ojdbc7.jar","--executor-memory","10g","--driver-memory","10g","--conf","spark.hadoop.yarn.timeline-service.enabled=false","--conf","currDate="Fri
Where am I going wrong?

You can use JSONLint to validate your JSON, which makes it easier to see why its wrong.
In your case, you are wrapping the final 3 values in single quotes ' rather than double quotes "
Your steps JSON should look like:
[{
"Args": [
"spark-submit",
"--deploy-mode",
"cluster",
"--jars",
"s3://tvsc-lumiq-edl/jars/ojdbc7.jar",
"--executor-memory",
"10g",
"--driver-memory",
"10g",
"--conf",
"spark.hadoop.yarn.timeline-service.enabled=false",
"--conf",
"currDate='\"$5\"'",
"--class",
"\"$2\"",
"\"$3\"",
"\"$4\""
],
"Type": "CUSTOM_JAR",
"ActionOnFailure": "CONTINUE",
"Jar": "command-runner.jar",
"Properties": "",
"Name": "Spark application"
}]
Specifically, these 3 lines:
"\"$2\"",
"\"$3\"",
"\"$4\""
Instead of the original:
'\"$2\"',
'\"$3\"',
'\"$4\"'

Visual Studio Code tasks.json issue with path in powershell

I have an addon that runs a command everytime I hit save.
I get the following error when I try to run a batch file on save:
The command autopush.bat was not found, but does exist in the current
location. Windows PowerShell does not load commands from the current
location by default. If you trust this command, instead type:
".\autopush.bat". See "get-help about_Command_Precedence" for more
details.
My settings.json file:
{
"saveAndRun": {
"commands": [
{
"match": ".*",
"cmd": "autopush.bat",
"useShortcut": false,
"silent": false
}
]
}
}
I am not sure about how I can add a dot and backslash to "autostart.bat"
I tried with ".\b autostart.bat" but it didn't work.

My bad, I used the wrong escape character. I had to use .\\autostart.bat as described here: How to escape special characters in building a JSON string?
\b Backspace (ascii code 08)
\f Form feed (ascii code 0C)
\n New line
\r Carriage return
\t Tab
\" Double quote
\\ Backslash character

Why doesn't Elasticsearch Ingest accept a grok pattern that Logstash does?

I have the following grok pattern that works in Logstash and in the Grok debugger in Kibana.
\[%{TIMESTAMP_ISO8601:req_time}\] %{IP:client_ip} (?:%{IP:forwarded_for}|\(-\)) (?:%{QS:request}|-) %{NUMBER:response_code:int} %{WORD}:%{NUMBER:request_length:int} %{WORD}:%{NUMBER:body_bytes_sent:int} %{WORD}:(?:%{QS:http_referer}|-) %{WORD}:(?:%{QS:http_user_agent}|-) (%{WORD}:(\")?(%{NUMBER:request_time:float})(\")?)?"
I am trying to create a new ingest pipeline via the PUT method, but I get an error that contains:
"type": "parse_exception",
"reason": "Failed to parse content to map",
"caused_by": {
"type": "i_o_exception",
"reason": "Unrecognized character escape '[' (code 91)\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper#61326735; line: 7, column: 25]"
}

Elasticsearch requires that grok patterns used in pipelines submitted using the PUT method are properly escaped JSON, while Logstash patterns use different escaping.
That includes preceding brackets with double backslashes (\\[) and double quotes with triple backslashes (\\\"). The working pattern (after running through a JSON escaping tool) is:
\\[%{TIMESTAMP_ISO8601:req_time}\\] %{IP:client_ip} (?:%{IP:forwarded_for}|\\(-\\)) (?:%{QS:request}|-) %{NUMBER:response_code:int} %{WORD}:%{NUMBER:request_length:int} %{WORD}:%{NUMBER:body_bytes_sent:int} %{WORD}:(?:%{QS:http_referer}|-) %{WORD}:(?:%{QS:http_user_agent}|-) (%{WORD}:(\\\")?(%{NUMBER:request_time:float})(\\\")?)?