I'm trying to extract some information from a nested JSON in log analytics.
It is nested several levels deep though. And I come unstuck at the 3rd tier.
The scenario is to query on what user ID has had permissions removed in Azure. the ifnormation is all there is raw format, but I want to extract it to be more readable.
The data layout is :
AzureActivity
Properties_d
responseBody
properties
principalId
It's the principalID I want (getting a UPN from AAD comes later ;)
My query works to a point. But the _propertieslevel3 comes up blank (no error).
_resonsebody is fine. It is a dynamic JSON that contains the responsebody field from Properties_d.
AzureActivity
| where (OperationNameValue contains "ROLEASSIGNMENTS/DELETE" and ActivityStatusValue contains "SUCCESS")
| extend _responsebody = parse_json(Properties_d.responseBody)
| extend _propertieslevel3 = parse_json(_responsebody.properties)
| extend ModifiedUser = parse_json(_propertieslevel3.principalId)
as _propertieslevel3 comes back blank, so does modified user. I can only guess that there is a problem trying to nest this deep.
Any ideas?
TIA.
data sample of Properties_d
{"eventCategory":"Administrative",
"eventDataId":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"eventSubmissionTimestamp":"2022-03-09T16:53:26.4493278Z",
"resource":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"resourceProviderValue":"MICROSOFT.AUTHORIZATION",
"subscriptionId":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"activityStatusValue":"Success",
"entity":"/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxx/providers/Microsoft.Authorization/roleAssignments/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"message":"Microsoft.Authorization/roleAssignments/delete",
"hierarchy":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"caller":"xxxxxx#xxxxxxx",
"httpRequest":"{\"clientIpAddress\":\"3.3.3.3\"}",
"statusCode":"OK",
"serviceRequestId":"",
"activitySubstatusValue":"OK",
"responseBody":"{\"properties\":{\"roleDefinitionId\":\"/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\",
\"principalId\":\"xxxxxxxxxxxxxxxxxxxxxxxxxxxx\",
\"principalType\":\"User\",
\"scope\":\"/subscriptions/xxxxxxxxxxxxxxxxxxxxxx\",
\"condition\":null,
\"conditionVersion\":null,
\"createdOn\":\"2022-03-09T11:28:48.4781104Z\",
\"updatedOn\":\"2022-03-09T11:28:48.4781104Z\",
\"createdBy\":\"xxxxxxxxxxxxxxxxxxxxxxxxx\",
\"updatedBy\":\"xxxxxxxxxxxxxxxxxxxxxxx\",
\"delegatedManagedIdentityResourceId\":null,
\"description\":null},
\"id\":\"/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/providers/Microsoft.Authorization/roleAssignments/xxxxxxxxxxxxxxxxxxxxxx\",
\"type\":\"Microsoft.Authorization/roleAssignments\",
\"name\":\"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"}"}
Most likely, you need to apply parse_json() on the nested property bag too.
See: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/parsejsonfunction
got it :)
Not sure why I needed to make _propertieslevel3 be just be the same as response body, rather than being able to extract .properties
but it works.
Thanks.
AzureActivity
| where (OperationNameValue contains "ROLEASSIGNMENTS/WRITE" and ActivityStatusValue contains "Start")
| extend _responsebody = parse_json(Properties_d.responseBody)
| extend _propertieslevel3 = parse_json(tostring(parse_json(_responsebody)))
| extend _level4 = parse_json(tostring(parse_json(_propertieslevel3.properties)))
| extend ModifiedUser = parse_json(tostring(parse_json(_level4.principalId)))
Related
I have an object in my database following a file upload that look like this
a:1:{s:4:"file";a:3:{s:7:"success";b:1;s:8:"file_url";a:2:{i:0;s:75:"http://landlordsplaces.com/wp-content/uploads/2021/01/23192643-threepersons.jpg";i:1;s:103:"http://landlordsplaces.com/wp-content/uploads/2021/01/364223-two-female-stick-figures.jpg";}s:9:"file_path";a:2:{i:0;s:93:"/var/www/vhosts/landlordsplaces.com/httpdocs/wp-content/uploads/2021/01/23192643-threepersons.jpg";i:1;s:121:"/var/www/vhosts/landlordsangel.com/httpdocs/wp-content/uploads/2021/01/364223-two-female-stick-figures.jpg";}}}
I am trying with no success to parse extract the two jpg urls programmatically from the object so i can show the images ont he site. Tried assigning parse(object) but that isnt helping. I just need to get the urls out.
Thank you in anticipation of any general direction
What you're looking at is not a JSON string. It is a serialized PHP object. If this database entry was created by Forminator, you should use the Forminator API to retrieve the needed form entry. The aforementioned link points to the get_entry method, which I suspect is what you're looking for (I have never used Forminator), but in any case, you should look for a method that will return that database entry as a PHP object containing your needed URLs.
In case it is ever of any help to anyone the answer to the question was based on John input. The API has the classes to handle that without needing to understand the data structure.
Forminator_API::initialize();
$form_id = 1449; // ID of a form
$entry_id = 3; // ID of an entry
$entry = Forminator_API::get_entry( $form_id, $entry_id );
$file_url = $entry->meta_data['upload-1']['value']['file']['file_url'];
$file_path = $entry->meta_data['upload-1']['value']['file']['file_path'];
var_dump($entry); //contains paths and urls
Hope someone benefits.
So, like the title says, I'm using Talend ESB in order to handle camel messaging. In my case, I'm sending the contents of a file as the message body to the child Talend job. In some scenarios the contents of the file may have 2+ rows. All I need is to be able to iterate over each of those rows independently within the child-job itself.
I guess my question is 2 folded. 1. If possible how do I do this? and 2. is the iteration process better suited at the route level, or the child-job the route calls.
Right now, the files I'm handling are | delimited. To handle this, I have the tRouteInput_1 going directly to a tExtractDelimtedFields and use those values to set variables globally, like so.
The problem with this, is it's only reading the first row of the file, and moving on. I need to be able to iterate over each row within the file/camel message.
Thanks,
Alex
First you need to split your file on the row delimiter using a tNormalize.
In my example, I simulate your tRouteInput by using a tFixedFlowInput containing the whole file as a single line, with rows separated by \n. Then for each resulting row returned by tNormalize, extract the fields you want (in tExtractDelimitedFields, create the schema corresponding to your row structure):
And the result:
.--------+--------.
| tLogRow_1 |
|=-------+-------=|
|field1 |field2 |
|=-------+-------=|
|field1.1|field1.2|
|field2.1|field2.2|
|field3.1|field3.2|
'--------+--------'
You need to escape "|" by using "\\|" inside tExtractDelimitedFields, as the component accepts regex, and the pipe has special meaning.
As for your 2nd question, I think it's better to do this inside the child job and not the route, as there are dedicated components for this not available in the routing perspective.
I have a Windows Store application which manages collection of objects and stores them in the application local folder. Those objects are serialized on the file system using JSON. As I need to be able to edit and persist those items individually I opted for individual files for each objects instead of one large file. Objects are stored following this pattern:
Local Folder
|
--- db
|
--- AB283376-7057-46B4-8B91-C32E663EC964
| |
| --- AB283376-7057-46B4-8B91-C32E663EC964.json
| --- AB283376-7057-46B4-8B91-C32E663EC964.jpg
|
--- B506EFC5-E853-45E6-BA32-64193BB49ACD
| |
| --- B506EFC5-E853-45E6-BA32-64193BB49ACD.json
| --- B506EFC5-E853-45E6-BA32-64193BB49ACD.jpg
|
...
Each object has its folder node which will contains the JSON serialized object and other eventual resources.
Everything was fine when I made some writing, reading, deleting test. Where it got complicated is when I tried to load up large collections of object on application startup. I estimated that the largest amount of item one would store to 10000. So I wrote 10000 entries and then tried to load it... more than 3 minutes to the application to complete the operation, which of course is unacceptable.
So my questions are, What could be optimized in the code I made for reading and deserializing objects (code below)? Is there a way to implement a paging system so loading would be dynamic in my WinRT application? Is my storage method (pattern above) too heavy for in terms of IO/CPU? Am I missing something in WinRT?
public async Task<IEnumerable<Release>> GetReleases()
{
List<Release> items = new List<Release>();
var dbFolder = await ApplicationData.Current.LocalFolder.CreateFolderAsync(dbName, CreationCollisionOption.OpenIfExists);
foreach (var releaseFolder in await dbFolder.GetFoldersAsync())
{
var releaseFile = await releaseFolder.GetFileAsync(releaseFolder.DisplayName + ".json");
var stream = await releaseFile.OpenAsync(FileAccessMode.Read);
using (var inStream = stream.GetInputStreamAt(0))
{
DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(Release));
Release release = (Release)serializer.ReadObject(inStream.AsStreamForRead());
items.Add(release);
}
stream.Dispose();
}
return items;
}
Thanks for your help.
NB: I already had a look as SQLite and I don't need such a sophisticated system.
Supposedly JSON.NET is better than the built in things. If you are not sending the data over the wire, then the quickest way is to do binary serialization rather than JSON or XML. Finally - think if you really need to load all the data when your application starts. Serialize your data as a list of binary records and create an index that will allow you to quickly jump to the range of records you actually need to use.
As Filip already mentioned, you probably don't need to load all data at startup. Even if you really want to show all the items in the first page (showing 10,000 items at once to a user doesn't sound like a good idea to me), you don't need to have all their properties available: usually only a couple of them are shown in the list, you need the rest of them when the user navigates to individual item details. You could have a separate "index" file containing only the data you need for the list. This does mean duplication, but it will help you with performance.
Although you've mentioned, you don't need SQLite as it is too sophisticated for your needs, you really should take a closer look at it. It is designed to efficiently handle structured data such as yours. I'm pretty sure if you switch to it, the performance will be much better and your code might end up even simpler in the end. Try it out.
I am putting together a parts database using the method below for directed acyclic graphs.
http://www.codeproject.com/Articles/22824/A-Model-to-Represent-Directed-Acyclic-Graphs-DAG-o
I am able to build my data set using the SQL queries from that page which I have converted to MySQL.
Previously I have used the nested sets model although we found that deletions became a problem.
I am unable to find any information on how to traverse the tree using this model. I simply need to be able to create a html tree to show the descendants from a selected parent node and identify leaf nodes (will be using jstree).
I can post the code from the nested sets model if that helps. I don't need any help with the html it is the SQL I am stuck with.
Does anyone have any idea where I can find information on the query I need.
EDIT:
Following on from the commments I'd like to adapt to something more closely linked to Bill Karwins closure model. http://www.slideshare.net/billkarwin/models-for-hierarchical-data
I notice however that on slides 49-50 which is where I want to select the descendants of a node that the output doesn't seem to provide enough to draw a simple tree. Previously with the nested sets model I was able to get a similar output that would traverse left to right, top to bottom. I'll try to explain.
Item | Depth
1 | 0
2 | 0
3 | 1
6 | 2
7 | 0
9 | 1
This allowed me to draw a tree as the SQL listed the order of descendants in a more manipulatable way. I believe it created "depth" by using a COUNT of subtrees and I will dig out the query if it would be useful here.
Thanks again for all your help.
I've just started to read upon graph-teory and data structures.
I'm building an example application which should be able to find the xpath for the most common links. Imagine a Google serp, my application should be able to find the xpath for all links pointing to a result.
Imagine that theese xpaths were found:
/html/body/h2/a
/html/body/p/a
/html/body/p/strong/a
/html/body/p/strong/a
/html/body/p/strong/a
/html/body/div[#class=footer]/span[#id=copyright]/a
From these xpats, i've thought of a graph like this (i might be completely lost here):
html
|
body
h2 - p - div[#class=footer]
| | |
a (1) a - strong span[#id=copyright]
| |
a (3) a (1)
Is this the best approach to this problem?
What would be the best way (data structure) to store this in memory? The language does not mather. We can see that we have 3 links matching the path html -> body -> p -> strong -> a.
As I said, i'm totally new to this so please forgive me if I thought of this completely wrong.
EDIT: I may be looking for the trie data structure?
Don't worry about tries yet. Just construct a tree using standard graph representation (node = {value, count, parent} while immediately collapsing same branches and incrementing the counter. Then, sort all the leaves by count in descending order and traverse from each leaf upwards to get a path.