Weird error while using externaldata operator in kusto query - json

I am trying to load the AWS ip ranges from their provided API as seen in the query below. If I use the proper url "https://ip-ranges.amazonaws.com/ip-ranges.json", I get the error message shown below. But if I run the same query for the json hosted at my own url "https://raw.githubusercontent.com/anandmudgerikar/aws-ips/main/ip-ranges.json", the query works fine. Any idea what might be happening? Thanks in advance.
Kusto Query:
externaldata(syncToken:string, createDate:string, prefixes: dynamic , ipv6_prefixes: dynamic)
[
h#'https://ip-ranges.amazonaws.com/ip-ranges.json'
//h#'https://api.github.com/users/anandmudgerikar/repos'
// h#'https://reqres.in/api/product/3'
//h#'https://www.dropbox.com/s/24117ufuyfanmew/ip-ranges.json'
//h#'https://raw.githubusercontent.com/anandmudgerikar/aws-ips/main/ip-ranges.json'
]
with(format= 'multijson', ingestionMapping=
'[{"Column":"syncToken","Properties":{"Path":"$.syncToken"}},'
'{"Column":"createDate","Properties":{"Path":"$.createDate"}},'
'{"Column":"prefixes","Properties":{"Path":"$.prefixes"}},'
'{"Column":"ipv6_prefixes","Properties":{"Path":"$.ipv6_prefixes"}}]')
This is the error message I get:
Query execution has resulted in error (0x80004003): Partial query failure: Invalid pointer (message: 'Argument 'name' is null: at .ctor in C:\source\Src\Common\Kusto.Cloud.Platform.Azure\Storage\PersistentStorage\BlobPersistentStorageFile.cs: line 55
Parameter name: name: ', details: 'Source: Kusto.Cloud.Platform
System.ArgumentNullException: Argument 'name' is null: at .ctor in C:\source\Src\Common\Kusto.Cloud.Platform.Azure\Storage\PersistentStorage\BlobPersistentStorageFile.cs: line 55
Parameter name: name
at Kusto.Cloud.Platform.Utils.Ensure.FailNullOrEmpty(String value, String argName, String callerMemberName, String callerFilePath, Int32 callerLineNumber) in C:\source\Src\Common\Kusto.Cloud.Platform\Diagnostics\Ensure.cs:line 150
at Kusto.Cloud.Platform.Azure.Storage.PersistentStorage.BlobPersistentStorageFile..ctor(CloudBlobContainer blobContainer, String name, IPersistentStorageFileCompressor persistentStorageFileCompressor, IPersistentStorageUri persistentStorageUri, TriState validBlobStorage, FileKnownMetadata knownMetadata) in C:\source\Src\Common\Kusto.Cloud.Platform.Azure\Storage\PersistentStorage\BlobPersistentStorageFile.cs:line 56
at Kusto.Cloud.Platform.Azure.Storage.PersistentStorage.BlobPersistentStorageFactory.CreateFileRef(String uri, IKustoTokenCredentialsProvider credentialsProvider, String compressionType, IPersistentStorageFileCompressorFactory persistentStorageFileCompressorFactory, StorageItemLocationMode locationMode, FileKnownMetadata knownMetadata) in C:\source\Src\Common\Kusto.Cloud.Platform.Azure\Storage\PersistentStorage\BlobPersistentStorageFactory.cs:line 214
at Kusto.Cloud.Platform.Storage.PersistentStorage.PersistentStorageFactoryFactory.CreateFileRef(String uri, IKustoTokenCredentialsProvider credentialsProvider, String compressionType, IPersistentStorageFileCompressorFactory persistentStorageFileCompressorFactory, StorageItemLocationMode locationMode, FileKnownMetadata knownMetadata) in C:\source\Src\Common\Kusto.Cloud.Platform\Storage\PersistentStorage\PersistentStorageFactoryFactory.cs:line 154
at Kusto.DataNode.DataEngineQueryPlan.ExternalDataQueryUtils.ReadExternalDataAsCsv(ArtifactEntry artifactEntry, DataSourceStreamFormat format, ExternalDataQueryCallbackContext callbackContext, Int64 recordCountLimit, String& columnsMapping) in C:\source\Src\Engine\DataNode\QueryService\DataEngineQueryPlan\ExternalDataQueryUtils.cs:line 101
at Kusto.DataNode.DataEngineQueryPlan.DataEngineQueryProcessor.DataEngineQueryCallback.GetExternalData(String externalDataUri, DataSourceStreamFormat format, String serializedCallbackContext, Int64 recordCountLimit) in C:\source\Src\Engine\DataNode\QueryService\DataEngineQueryPlan\DataEngineQueryProcessor.cs:line 399').
clientRequestId: KustoWebV2;71c590e6-27ab-41c9-bf3c-03ba4ee0cf3b

Only storage connection strings that are documented here are officially supported; There are no guarantees for others.
To comply with that, for example, you can copy the file from its original location to an Azure blob container, and query it from there.

Related

How do you display a specific phrase your looking for from a string in POSTGRESQL

Here is the sample row value
CREATION_DATE: 2021-08-30
USER_EMAIL: 'jack#sample.com
ACTION: 'FileAccessedExtended'
AUDITDATA:
{
"OrganizationId":"orgid1231231",
"RecordType":6,
"UserKey":"i:0h.f|membership|10032000de9e9feb#live.com",
"UserType":0,
"Version":1,
"Workload":"SharePoint",
"ClientIP":"11.11.11.11",
"ObjectId":"https:\/\/someorganization.sharepoint.com\/sites\/MondayMeetings862\/Shared Documents\/folder\/folders\/somefolder\/Contact Database\/Investment Contact Database (MASTER) 1.xlsx",
"UserId":"jack#sample.com",
"CorrelationId":"12-031-23809123809172309",
"EventSource":"SharePoint",
"ItemType":"File",
"ListId":"334b5378-2a4c-4ba6-b1ac-1ccb51a38687",
"ListItemUniqueId":"a46a8f64-0ecf-415b-b372-32852c160d74",
"Site":"2839182391823819238192389",
"UserAgent":"MSOCS",
"WebId":"1092301723729813",
"HighPriorityMediaProcessing":false,
"SourceFileExtension":"xlsx",
"SiteUrl":"https:\/\/someorganization.sharepoint.com\/sites\/MondayMeetings862\/",
"SourceFileName":"Investment Contact Database (MASTER) 1.xlsx",
"SourceRelativeUrl":"Shared Documents\/folder\/folders\/COMMERCIAL\/Contact Database"
}
Desired output:
CREATION_DATE: 2021-08-30
USER_EMAIL: Jack#sample.com
ACTION: File Access Extended
CLIENTIP: 11.11.11.11
Assuming auditdata is a column of type jsonb (or at least json) you can use the ->> operator to extract a value from it:
select creation_date, user_email, action,
auditdata ->> 'ClientIP' as client_ip
from the_table;
If for some reason, the column is defined properly as a jsonb (or at least json) you need to cast the value auditdata::jsonb ->> 'ClientIP'
Ok. Here come a bunch of if's. If AuditData is an array. And if ClientIP is reliably in the same position, then
SELECT AuditData[6] FROM MyTable
should give
"ClientIP":"11.11.11.11"
You can remove the double quotes by using the string function replace to replace double quote with nothing.

Cannot resolve keyword 'post_id' into field. Choices are: comment, id, name, post_list

FieldError at /posts/1/
Cannot resolve keyword 'post_id' into field. Choices are: comment, id, name, post_list
Request Method: GET
Request URL: http://127.0.0.1:8000/posts/1/
Django Version: 3.1.4
Exception Type: FieldError
Exception Value:
Cannot resolve keyword 'post_id' into field. Choices are: comment, id, name, post_list
Exception Location: C:\Users\Emiedonmokumo\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\sql\query.py, line 1481, in names_to_path
Python Executable: C:\Users\Emiedonmokumo\AppData\Local\Programs\Python\Python39\python.exe
Python Version: 3.9.0
Python Path:
['C:\\Users\\Emiedonmokumo\\Documents\\mysite\\admin',
'C:\\Users\\Emiedonmokumo\\AppData\\Local\\Programs\\Python\\Python39\\python39.zip',
'C:\\Users\\Emiedonmokumo\\AppData\\Local\\Programs\\Python\\Python39\\DLLs',
'C:\\Users\\Emiedonmokumo\\AppData\\Local\\Programs\\Python\\Python39\\lib',
'C:\\Users\\Emiedonmokumo\\AppData\\Local\\Programs\\Python\\Python39',
'C:\\Users\\Emiedonmokumo\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages']
Server time: Sun, 13 Dec 2020 11:42:03 +0000
there keep directing me to this line of code: c_number = len(Comment.objects.filter(post_id=pk)) …
please help me out, thanks
As the error indicates, there is no such field as post_id, there are only comment, id, name and post_list.
Likely you want to filter on post_list, so then the query should be:
c_number = len(Comment.objects.filter(post_list=pk))
Using len(…) is however not a good idea, since that will load all recors in memory and then obtain the length, this thus is inefficient use of bandwidth. You can make use of .count() [Django-doc] to count the number of records at the database side:
c_number = Comment.objects.filter(post_list=pk).count()

Azure Data Factory - MS Access as Source Database - Error

My source is 'Access Database'
Dynamically generating Source query as 'Select * from <tableName>'
But I got field names with spaces in source table, and destination is of type .parquet, Data Factory pipeline is failing with below error
Example if Table Employee got a column 'First Name'
{
"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=UserErrorJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java,
message: java.lang.IllegalArgumentException:field ended by ';': expected ';' but got 'Area' at line 0: message adms_schema { optional binary RAM Area\ntotal entry:10\r
\norg.apache.parquet.schema.MessageTypeParser.check(MessageTypeParser.java:215)\r\norg.apache.parquet.schema.MessageTypeParser.addPrimitiveType(MessageTypeParser.java:188)\r
\norg.apache.parquet.schema.MessageTypeParser.addType(MessageTypeParser.java:112)\r\norg.apache.parquet.schema.MessageTypeParser.addGroupTypeFields(MessageTypeParser.java:100)\r
\norg.apache.parquet.schema.MessageTypeParser.parse(MessageTypeParser.java:93)\r\norg.apache.parquet.schema.MessageTypeParser.parseMessageType(MessageTypeParser.java:83)\r
\ncom.microsoft.datatransfer.bridge.parquet.ParquetWriterBuilderBridge.getSchema(ParquetWriterBuilderBridge.java:187)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetWriterBuilderBridge.build
(ParquetWriterBuilderBridge.java:159)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetWriterBridge.open(ParquetWriterBridge.java:13)\r
\ncom.microsoft.datatransfer.bridge.parquet.ParquetFileBridge.createWriter(ParquetFileBridge.java:27)\r
\n,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'",
"failureType": "UserError",
"target": "Copy Table Data to Sink",
"details": []
}
if i change query to SELECT [First Name] as FirstName from Employee, it works fine.
As am generating query dynamically, i was using '*'
Is there some setting on Sink (.parquet) to ignore spaces in column names?
EDIT some info here https://issues.apache.org/jira/browse/SPARK-4521, not sure how to deal in ADF.
And this link: https://github.com/MicrosoftDocs/azure-docs/issues/28320

Rails with mysql can't save json datatype

I have create Rails(3.2) application with mysql(5.7.16) backend. I can't save json data and I have pass value like following and it shows error like
ActiveRecord::StatementInvalid - Mysql2::Error: Invalid JSON text: "Invalid value." at position 1 in value for column 'shopping_cart_item_special_infos.special_info'.: INSERT INTO shopping_cart_item_special_infos (created_at, shopping_cart_checkout_option_id, special_info, updated_at) VALUES ('2016-12-29 06:08:52', 141, '--- !ruby/hash:ActiveSupport::HashWithIndifferentAccess\n\'201\': text1\n\'209\': \'623\'\n\'210\': \'625\'\n\'211\': text2\n', '2016-12-29 06:08:52'):
activerecord (3.2.21) lib/active_record/connection_adapters/abstract_adapter.rb:285:in rescue in log'
activerecord (3.2.21) lib/active_record/connection_adapters/abstract_adapter.rb:280:inlog'
newrelic_rpm (3.9.1.236) lib/new_relic/agent/instrumentation/active_record.rb:63:in `block in log_with_newrelic_i
{"201"=>"text1", "209"=>"623", "210"=>"625", "211"=>"text2"}
What was wrong here ?
Your JSON should be like this
{"201":"text1", "209":"623", "210":"625", "211":"text2"}
But you are passing a ruby hash
Check the difference between => and :
And as #amruth-ls suggested you can validate your JSON at jsonlint
Eventually I fixed above error by to_json method
Ex)
{"201"=>"text1", "209"=>"623", "210"=>"625", "211"=>"text2"}.to_json
NOTE: I am used to_json because I am not serialised value as JSON using code like
serialize :special_info, JSON
If I used above code then no need to convert using as_json, it will automatically parse above data as json data otherwise it consider the data as string.

Escape quotes inside quoted fields when parsing CSV in Flink

In Flink, parsing a CSV file using readCsvFile raises an exception when encountring a field containing quotes like "Fazenda São José ""OB"" Airport":
org.apache.flink.api.common.io.ParseException: Line could not be parsed: '191,"SDOB","small_airport","Fazenda São José ""OB"" Airport",-21.425199508666992,-46.75429916381836,2585,"SA","BR","BR-SP","Tapiratiba","no","SDOB",,"SDOB",,,'
I've found in this mailing list thread and this JIRA issue that quoting inside the field should be realized through the \ character, but I don't have control over the data to modify it. Is there a way to work around this?
I've also tried using ignoreInvalidLines() (which is the less preferable solution) but it gave me the following error:
08:49:05,737 INFO org.apache.flink.api.common.io.LocatableInputSplitAssigner - Assigning remote split to host localhost
08:49:05,765 ERROR org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN DataSource (at main(Job.java:53) (org.apache.flink.api.java.io.TupleCsvInputFormat)) -> Map (Map at main(Job.java:54)) -> Combine(SUM(1), at main(Job.java:56) (2/8)
java.lang.ArrayIndexOutOfBoundsException: -1
at org.apache.flink.api.common.io.GenericCsvInputFormat.skipFields(GenericCsvInputFormat.java:443)
at org.apache.flink.api.common.io.GenericCsvInputFormat.parseRecord(GenericCsvInputFormat.java:412)
at org.apache.flink.api.java.io.CsvInputFormat.readRecord(CsvInputFormat.java:111)
at org.apache.flink.api.common.io.DelimitedInputFormat.nextRecord(DelimitedInputFormat.java:454)
at org.apache.flink.api.java.io.CsvInputFormat.nextRecord(CsvInputFormat.java:79)
at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:176)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
at java.lang.Thread.run(Thread.java:745)
Here is my code:
DataSet<Tuple2<String, Integer>> csvInput = env.readCsvFile("resources/airports.csv")
.ignoreFirstLine()
.ignoreInvalidLines()
.parseQuotedStrings('"')
.includeFields("100000001")
.types(String.class, String.class)
.map((Tuple2<String, String> value) -> new Tuple2<>(value.f1, 1))
.groupBy(0)
.sum(1);
If you cannot change the input data, then you should turn off parseQuotedString(). This will simply look for the next field delimiter and return everything in between as a string (including the quotations marks). Then you can remove the leading and trailing quotation mark in a subsequent map operation.