Import big XML into MySQL - mysql

I have to import the following XML file into MySQL. I can only get the row "Name" to work. The import doesn't work because on "Class", "Subclass", etc. there is an additional "Name" tag inside.
I also have to import the "PropertyData", but how do I do that?
The XML File can't be changed under any circumstances!
<MatML_Doc>
<Material>
<BulkDetails>
<Name>ABS</Name>
<Class> <Name>PLASTIC</Name></Class>
<Subclass> <Name>ABS Polymer</Name></Subclass>
<PropertyData property="Material_Type">
<Data format="string">IsotropicMaterial</Data>
</PropertyData>
<PropertyData property="Version">
<Data format="string">4.0</Data>
</PropertyData>
<PropertyData property="Category">
<Data format="string">PLASTIC</Data>
</PropertyData>
<PropertyData property="Mass_Density__RHO_NonIso__6">
<Data format="exponential">1.050000e-006</Data>
</PropertyData>
<PropertyData property="Mass_Density__RHO__6">
<Data format="exponential">1.050000e-006</Data>
</PropertyData>
<PropertyData property="Youngs_Modulus_E_NonIso__31">
<Data format="exponential">2.000000e+006</Data>
</PropertyData>
<PropertyData property="Youngs_Modulus_E__31">
<Data format="exponential">2.000000e+006</Data>
</PropertyData>
<PropertyData property="PoissonsRatio__1">
<Data format="exponential">4.000000e-001</Data>
</PropertyData>
<PropertyData property="PoissonsRatio">
<Data format="exponential">4.000000e-001</Data>
</PropertyData>
<PropertyData property="MatlNonlinearityType">
<Data format="integer">1</Data>
</PropertyData>
<PropertyData property="YieldFunctionCriterion">
<Data format="integer">1</Data>
</PropertyData>
<PropertyData property="HardeningRule">
<Data format="integer">1</Data>
</PropertyData>
<PropertyData property="Yield_Strength_32">
<Data format="exponential">4.000000e+004</Data>
</PropertyData>
<PropertyData property="Thermal_Expansion_A_NonIso__34">
<Data format="exponential">7.000000e-005</Data>
</PropertyData>
<PropertyData property="Thermal_Expansion_A__34">
<Data format="exponential">7.000000e-005</Data>
</PropertyData>
<PropertyData property="Thermal_Conductivity__K__35">
<Data format="exponential">1.700000e+002</Data>
</PropertyData>
<PropertyData property="Specific_Heat_CP__23">
<Data format="exponential">1.800000e+009</Data>
</PropertyData>
<PropertyData property="Crosshatch_Pattern">
<Data format="string">Rubber/Plastic</Data>
</PropertyData>
<PropertyData property="CoatingsStudioMaterialName">
<Data format="string">ABS Plastic</Data>
</PropertyData>
<PropertyData property="CoatingsVisualizationColor">
<Data format="exponential">168, 168, 168</Data>
</PropertyData>
</BulkDetails>
</Material> FILE CONTINUES....
My Code so far:
DROP DATABASE IF EXISTS testdb;
CREATE DATABASE testdb;
USE testdb;
CREATE TABLE bulkdetails (
Name VARCHAR(40),
Class VARCHAR(40),
Subclass VARCHAR(40));
LOAD XML INFILE "C:\physicalmateriallibrary.xml"
INTO TABLE bulkdetails
ROWS IDENTIFIED BY '<BulkDetails>';

Related

Graph DB Cluster identification followed by Cluster Comparison with externally defined Clusters

I have a commercially sensitive system, so I can't describe it directly but will try to use an obfuscated data set and description.
This a question about what analyses are available to run, not how to program it. If you're feeling flamey about 'how' or 'where' I've asked the question, please be a cool human and just point me to the right place to post.
For context, I'm running Gremlin server 3.4.6 in Docker on a Debian VM and Gremlin Console 3.4.4 on a Mac and I'm hoping to import my data into the server from a graphml file that successfully renders in Gephy.
I have
6 'widget types'
30 'Things'
175 'people'
In world A
Each thing is atype of widget
[Each widget type has one or more instance Thing]
Each Thing is used by zero or more people
[Each person uses zero or more things]
In a parallel world, B there's a probably similar structure (we think is similar but we want to find out how similar clusters from each world actually are)
For the graph in world A, we run an analysis (like k-means) primarily to identify clusters of Things and people that work together. Having done that we do something similar to World B
I would then like to know if there's a graph analysis that will give me:
Confidence estimation of affinities between World A groups and World B groups
Snippet of the data (vast majority of data excluded. I have included 1 Type, 1 Thing, 1 Person and a few arbitrary edges):
<?xml version="1.0" encoding="UTF-8"?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns">
<key attr.name="label" attr.type="string" for="node" id="label"/>
<key attr.name="Edge Label" attr.type="string" for="edge" id="edgelabel"/>
<key attr.name="weight" attr.type="double" for="edge" id="weight"/>
<key attr.name="r" attr.type="int" for="node" id="r"/>
<key attr.name="g" attr.type="int" for="node" id="g"/>
<key attr.name="b" attr.type="int" for="node" id="b"/>
<key attr.name="x" attr.type="float" for="node" id="x"/>
<key attr.name="y" attr.type="float" for="node" id="y"/>
<key attr.name="size" attr.type="float" for="node" id="size"/>
<key attr.name="interval" attr.type="intervalset" for="node" id="interval"/>
<key attr.name="type" attr.type="string" for="node" id="type"/>
<key attr.name="Clustering Coefficient" attr.type="double" for="node" id="clustering"/>
<key attr.name="Number of triangles" attr.type="int" for="node" id="triangles"/>
<key attr.name="Eigenvector Centrality" attr.type="double" for="node" id="eigencentrality"/>
<key attr.name="Inferred Class" attr.type="int" for="node" id="stat_inf_class"/>
<key attr.name="Modularity Class" attr.type="int" for="node" id="modularity_class"/>
<key attr.name="Authority" attr.type="float" for="node" id="authority"/>
<key attr.name="Hub" attr.type="float" for="node" id="hub"/>
<key attr.name="Component ID" attr.type="int" for="node" id="componentnumber"/>
<key attr.name="Cluter-ID" attr.type="int" for="node" id="cluter-id"/>
<key attr.name="Cluster" attr.type="int" for="node" id="cluster"/>
<key attr.name="sourcelabel" attr.type="string" for="edge" id="sourcelabel"/>
<key attr.name="targetlabel" attr.type="string" for="edge" id="targetlabel"/>
<key attr.name="interval" attr.type="intervalset" for="edge" id="interval"/>
<graph edgedefault="undirected">
<node id="1">
<data key="label">WIDGET1</data>
<data key="type">Type/Class</data>
<data key="clustering">0.0</data>
<data key="triangles">0</data>
<data key="eigencentrality">0.0</data>
<data key="stat_inf_class">0</data>
<data key="modularity_class">0</data>
<data key="authority">0.0</data>
<data key="hub">0.0</data>
<data key="componentnumber">0</data>
<data key="cluter-id">0</data>
<data key="cluster">0</data>
<data key="size">10.0</data>
<data key="r">255</data>
<data key="g">26</data>
<data key="b">32</data>
<data key="x">-624.7099</data>
<data key="y">0.96178514</data>
</node>
<node id="7">
<data key="label">ABNDFILE</data>
<data key="type">Thing</data>
<data key="clustering">0.0</data>
<data key="triangles">0</data>
<data key="eigencentrality">0.08110029554883653</data>
<data key="stat_inf_class">12</data>
<data key="modularity_class">10</data>
<data key="authority">0.028190823</data>
<data key="hub">0.0070641674</data>
<data key="componentnumber">6</data>
<data key="cluter-id">19</data>
<data key="cluster">6</data>
<data key="size">10.0</data>
<data key="r">255</data>
<data key="g">26</data>
<data key="b">32</data>
<data key="x">-990.6759</data>
<data key="y">-714.0662</data>
</node>
<node id="46">
<data key="label">CRGN1AAA</data>
<data key="type">person</data>
<data key="clustering">0.0</data>
<data key="triangles">0</data>
<data key="eigencentrality">3.014088737416253E-4</data>
<data key="stat_inf_class">11</data>
<data key="modularity_class">8</data>
<data key="authority">0.0</data>
<data key="hub">0.0</data>
<data key="componentnumber">7</data>
<data key="cluter-id">7</data>
<data key="cluster">45</data>
<data key="size">10.0</data>
<data key="r">255</data>
<data key="g">26</data>
<data key="b">32</data>
<data key="x">828.32367</data>
<data key="y">-881.0575</data>
</node>
<edge id="1" label="JKL" source="46" target="15">
<data key="edgelabel">JKL</data>
<data key="weight">1.0</data>
<data key="sourcelabel">CRGN1AAA</data>
<data key="targetlabel">CWXN</data>
</edge>
<edge id="2" label="JKL" source="47" target="21">
<data key="edgelabel">JKL</data>
<data key="weight">1.0</data>
<data key="sourcelabel">CWEB1AAA</data>
<data key="targetlabel">DFHPISN2</data>
</edge>
<edge id="3" label="JKL" source="47" target="41">
<data key="edgelabel">JKL</data>
<data key="weight">1.0</data>
<data key="sourcelabel">CWEB1AAA</data>
<data key="targetlabel">SWITCHTX</data>
</edge>
</graph>
</graphml>

Access custom Increment ("yy" & 0000) [duplicate]

I am working on an Access 2013 database that will have different utility poles entered into the database and linked with other attributes. Each pole will have a unique global ID, and to simplify working I would like to add another unique ID that is more simple. I would like this field auto populated when a new pole in imported into the database. The ID would go as follows:
SAC(year)-(Escalating number, cannot be a duplicate)
ex. SAC16-20 (This would be the 20th pole entered into the database in 2016)
ex. SAC15-2536 (Would be the 2536th pole entered in 2015)
If anyone could help me generate some code to make this auto populate ID field work I would greatly appreciate it.
With Access versions 2010 and later you can use an event-driven data macro to generate the sequential ID. For example, say you have a table named [poledata]. Open it in Design View and add two fields:
alternate_id_seq – Numeric (Long Integer)
alternate_id – Text(20)
Save the changes to your table and then switch to Datasheet View.
In the Access ribbon, switch to the "Table Tools > Table" tab and click "Before Change"
then enter the following macro ...
... or paste the following XML into the macro editor window
<?xml version="1.0" encoding="utf-16" standalone="no"?>
<DataMacros xmlns="http://schemas.microsoft.com/office/accessservices/2009/11/application">
<DataMacro Event="BeforeChange">
<Statements>
<ConditionalBlock>
<If>
<Condition>[IsInsert]</Condition>
<Statements>
<Action Name="SetLocalVar">
<Argument Name="Name">next_seq</Argument>
<Argument Name="Value">1</Argument>
</Action>
<Action Name="SetLocalVar">
<Argument Name="Name">prefix</Argument>
<Argument Name="Value">"SAC" & Year(Date()) Mod 100 & "-"</Argument>
</Action>
<LookUpRecord>
<Data Alias="pd">
<Query>
<References>
<Reference Source="poledata" Alias="pd" />
</References>
<Results>
<Property Source="pd" Name="alternate_id_seq" />
</Results>
<Ordering>
<Order Direction="Descending" Source="pd" Name="alternate_id_seq" />
</Ordering>
</Query>
<WhereCondition>[pd].[alternate_id] Like [prefix] & "*"</WhereCondition>
</Data>
<Statements>
<Action Name="SetLocalVar">
<Argument Name="Name">next_seq</Argument>
<Argument Name="Value">[pd].[alternate_id_seq]+1</Argument>
</Action>
</Statements>
</LookUpRecord>
<Action Name="SetField">
<Argument Name="Field">alternate_id_seq</Argument>
<Argument Name="Value">[next_seq]</Argument>
</Action>
<Action Name="SetField">
<Argument Name="Field">alternate_id</Argument>
<Argument Name="Value">[prefix] & [next_seq]</Argument>
</Action>
</Statements>
</If>
</ConditionalBlock>
</Statements>
</DataMacro>
</DataMacros>
Now when new rows are added to the table the [alternate_id_seq] and [alternate_id] columns will be populated automatically.

How to autoincrease PO number , and link it to my main table? [duplicate]

I am working on an Access 2013 database that will have different utility poles entered into the database and linked with other attributes. Each pole will have a unique global ID, and to simplify working I would like to add another unique ID that is more simple. I would like this field auto populated when a new pole in imported into the database. The ID would go as follows:
SAC(year)-(Escalating number, cannot be a duplicate)
ex. SAC16-20 (This would be the 20th pole entered into the database in 2016)
ex. SAC15-2536 (Would be the 2536th pole entered in 2015)
If anyone could help me generate some code to make this auto populate ID field work I would greatly appreciate it.
With Access versions 2010 and later you can use an event-driven data macro to generate the sequential ID. For example, say you have a table named [poledata]. Open it in Design View and add two fields:
alternate_id_seq – Numeric (Long Integer)
alternate_id – Text(20)
Save the changes to your table and then switch to Datasheet View.
In the Access ribbon, switch to the "Table Tools > Table" tab and click "Before Change"
then enter the following macro ...
... or paste the following XML into the macro editor window
<?xml version="1.0" encoding="utf-16" standalone="no"?>
<DataMacros xmlns="http://schemas.microsoft.com/office/accessservices/2009/11/application">
<DataMacro Event="BeforeChange">
<Statements>
<ConditionalBlock>
<If>
<Condition>[IsInsert]</Condition>
<Statements>
<Action Name="SetLocalVar">
<Argument Name="Name">next_seq</Argument>
<Argument Name="Value">1</Argument>
</Action>
<Action Name="SetLocalVar">
<Argument Name="Name">prefix</Argument>
<Argument Name="Value">"SAC" & Year(Date()) Mod 100 & "-"</Argument>
</Action>
<LookUpRecord>
<Data Alias="pd">
<Query>
<References>
<Reference Source="poledata" Alias="pd" />
</References>
<Results>
<Property Source="pd" Name="alternate_id_seq" />
</Results>
<Ordering>
<Order Direction="Descending" Source="pd" Name="alternate_id_seq" />
</Ordering>
</Query>
<WhereCondition>[pd].[alternate_id] Like [prefix] & "*"</WhereCondition>
</Data>
<Statements>
<Action Name="SetLocalVar">
<Argument Name="Name">next_seq</Argument>
<Argument Name="Value">[pd].[alternate_id_seq]+1</Argument>
</Action>
</Statements>
</LookUpRecord>
<Action Name="SetField">
<Argument Name="Field">alternate_id_seq</Argument>
<Argument Name="Value">[next_seq]</Argument>
</Action>
<Action Name="SetField">
<Argument Name="Field">alternate_id</Argument>
<Argument Name="Value">[prefix] & [next_seq]</Argument>
</Action>
</Statements>
</If>
</ConditionalBlock>
</Statements>
</DataMacro>
</DataMacros>
Now when new rows are added to the table the [alternate_id_seq] and [alternate_id] columns will be populated automatically.

Create sequential ID value based on the year that a record is added

I am working on an Access 2013 database that will have different utility poles entered into the database and linked with other attributes. Each pole will have a unique global ID, and to simplify working I would like to add another unique ID that is more simple. I would like this field auto populated when a new pole in imported into the database. The ID would go as follows:
SAC(year)-(Escalating number, cannot be a duplicate)
ex. SAC16-20 (This would be the 20th pole entered into the database in 2016)
ex. SAC15-2536 (Would be the 2536th pole entered in 2015)
If anyone could help me generate some code to make this auto populate ID field work I would greatly appreciate it.
With Access versions 2010 and later you can use an event-driven data macro to generate the sequential ID. For example, say you have a table named [poledata]. Open it in Design View and add two fields:
alternate_id_seq – Numeric (Long Integer)
alternate_id – Text(20)
Save the changes to your table and then switch to Datasheet View.
In the Access ribbon, switch to the "Table Tools > Table" tab and click "Before Change"
then enter the following macro ...
... or paste the following XML into the macro editor window
<?xml version="1.0" encoding="utf-16" standalone="no"?>
<DataMacros xmlns="http://schemas.microsoft.com/office/accessservices/2009/11/application">
<DataMacro Event="BeforeChange">
<Statements>
<ConditionalBlock>
<If>
<Condition>[IsInsert]</Condition>
<Statements>
<Action Name="SetLocalVar">
<Argument Name="Name">next_seq</Argument>
<Argument Name="Value">1</Argument>
</Action>
<Action Name="SetLocalVar">
<Argument Name="Name">prefix</Argument>
<Argument Name="Value">"SAC" & Year(Date()) Mod 100 & "-"</Argument>
</Action>
<LookUpRecord>
<Data Alias="pd">
<Query>
<References>
<Reference Source="poledata" Alias="pd" />
</References>
<Results>
<Property Source="pd" Name="alternate_id_seq" />
</Results>
<Ordering>
<Order Direction="Descending" Source="pd" Name="alternate_id_seq" />
</Ordering>
</Query>
<WhereCondition>[pd].[alternate_id] Like [prefix] & "*"</WhereCondition>
</Data>
<Statements>
<Action Name="SetLocalVar">
<Argument Name="Name">next_seq</Argument>
<Argument Name="Value">[pd].[alternate_id_seq]+1</Argument>
</Action>
</Statements>
</LookUpRecord>
<Action Name="SetField">
<Argument Name="Field">alternate_id_seq</Argument>
<Argument Name="Value">[next_seq]</Argument>
</Action>
<Action Name="SetField">
<Argument Name="Field">alternate_id</Argument>
<Argument Name="Value">[prefix] & [next_seq]</Argument>
</Action>
</Statements>
</If>
</ConditionalBlock>
</Statements>
</DataMacro>
</DataMacros>
Now when new rows are added to the table the [alternate_id_seq] and [alternate_id] columns will be populated automatically.

Create a concatenated list of child table values from an Access database without using VBA

I have an Access 2010 database with a relationship between parent and child tables. I would like to be able to query the database from an external application and show values from the child table as a concatenated list of values in a single column, similar to what MySQL can produce with its GROUP_CONCAT() function.
This has been asked here many times before, e.g., here:
Combine values from related rows into a single concatenated string value
but those solutions rely on a custom VBA function that is not available to external queries.
Is there a way to make such a concatenated list available to external queries without having to manually build the list in the other application?
Issue
Historically, the Access solution for a GROUP_CONCAT()-type query has been to use a VBA function like Allen Browne's ConcatRelated() (ref: here). However, custom VBA functions are only available to queries run from within Microsoft Access itself, so this is not a viable solution for queries against an Access database from some other application (e.g., a .NET application using OLEDB or ODBC).
Solution
With an Access 2010 (or newer) database we can emulate the behaviour of a MySQL GROUP_CONCAT() query by adding a Long Text ("Memo") field to the parent table and using data macros on the child table to maintain the concatenated list.
For example, for tables [Parents] ...
ParentID ParentName
-------- -------------
1 Homer Simpson
2 Ned Flanders
... and [Children] ...
ChildID ParentID ChildName DisplayOrder
------- -------- ----------------- ------------
1 1 Lisa 2
2 1 Bart 1
3 2 Rod, the elder 1
4 1 Maggie 3
5 2 Todd, the younger 2
... we can add a new Memo/Long Text field named [ChildList] to the [Parents] table and then add the following data macros to the [Children] table:
[Named Macro: UpdateChildList]
<?xml version="1.0" encoding="UTF-16" standalone="no"?>
<DataMacros xmlns="http://schemas.microsoft.com/office/accessservices/2009/11/application">
<DataMacro Name="UpdateChildList">
<Parameters>
<Parameter Name="prmParentID"/>
</Parameters>
<Statements>
<Action Collapsed="true" Name="SetLocalVar">
<Argument Name="Name">newList</Argument>
<Argument Name="Value">Null</Argument>
</Action>
<ForEachRecord>
<Data Alias="c">
<Query>
<References>
<Reference Source="Children" Alias="c"/>
</References>
<Results>
<Property Source="c" Name="ChildName"/>
</Results>
<Ordering>
<Order Source="c" Name="DisplayOrder"/>
</Ordering>
</Query>
<WhereCondition>[c].[ParentID]=[prmParentID] And [c].[ChildName] Is Not Null</WhereCondition>
</Data>
<Statements>
<ConditionalBlock>
<If>
<Condition>Not IsNull([newList])</Condition>
<Statements>
<Action Collapsed="true" Name="SetLocalVar">
<Argument Name="Name">newList</Argument>
<Argument Name="Value">[newList] & ";" & Chr(160)</Argument>
</Action>
</Statements>
</If>
</ConditionalBlock>
<Action Collapsed="true" Name="SetLocalVar">
<Argument Name="Name">newList</Argument>
<Argument Name="Value">[newList] & [c].[ChildName]</Argument>
</Action>
</Statements>
</ForEachRecord>
<LookUpRecord>
<Data>
<Reference>Parents</Reference>
<WhereCondition>[Parents].[ParentID]=[prmParentID]</WhereCondition>
</Data>
<Statements>
<EditRecord>
<Data/>
<Statements>
<Action Collapsed="true" Name="SetField">
<Argument Name="Field">Parents.ChildList</Argument>
<Argument Name="Value">[newList]</Argument>
</Action>
</Statements>
</EditRecord>
</Statements>
</LookUpRecord>
</Statements>
</DataMacro>
</DataMacros>
[After Insert]
<?xml version="1.0" encoding="UTF-16" standalone="no"?>
<DataMacros xmlns="http://schemas.microsoft.com/office/accessservices/2009/11/application">
<DataMacro Event="AfterInsert">
<Statements>
<Action Name="RunDataMacro">
<Argument Name="MacroName">Children.UpdateChildList</Argument>
<Parameters>
<Parameter Name="prmParentID" Value="[ParentID]"/>
</Parameters>
</Action>
</Statements>
</DataMacro>
</DataMacros>
[After Update]
<?xml version="1.0" encoding="UTF-16" standalone="no"?>
<DataMacros xmlns="http://schemas.microsoft.com/office/accessservices/2009/11/application">
<DataMacro Event="AfterUpdate">
<Statements>
<ConditionalBlock>
<If>
<Condition>Updated("ParentID") Or Updated("ChildName")</Condition>
<Statements>
<Action Name="RunDataMacro">
<Argument Name="MacroName">Children.UpdateChildList</Argument>
<Parameters>
<Parameter Name="prmParentID" Value="[ParentID]"/>
</Parameters>
</Action>
<ConditionalBlock>
<If>
<Condition>Updated("ParentID")</Condition>
<Statements>
<Action Name="RunDataMacro">
<Argument Name="MacroName">Children.UpdateChildList</Argument>
<Parameters>
<Parameter Name="prmParentID" Value="[Old].[ParentID]"/>
</Parameters>
</Action>
</Statements>
</If>
</ConditionalBlock>
</Statements>
</If>
</ConditionalBlock>
</Statements>
</DataMacro>
</DataMacros>
[After Delete]
<?xml version="1.0" encoding="UTF-16" standalone="no"?>
<DataMacros xmlns="http://schemas.microsoft.com/office/accessservices/2009/11/application">
<DataMacro Event="AfterDelete">
<Statements>
<Action Name="RunDataMacro">
<Argument Name="MacroName">Children.UpdateChildList</Argument>
<Parameters>
<Parameter Name="prmParentID" Value="[Old].[ParentID]"/>
</Parameters>
</Action>
</Statements>
</DataMacro>
</DataMacros>
Results
As changes are made to the child table the list in the parent table will automatically be updated:
ParentID ParentName ChildList
-------- ------------- ---------------------------------
1 Homer Simpson Bart; Lisa; Maggie
2 Ned Flanders Rod, the elder; Todd, the younger
Notes
The [ChildList] field is for display purposes only. Editing the values in that field will not change the values in the child table.
The list is separated with ";" & Chr(160) to differentiate it from any ";" & Chr(32) pairs that may be in the actual data. If the non-breaking space (Chr(160)) characters mess up wrapping of the list then we could use the Replace() function in our query to convert ";" & Chr(160) to ";" & Chr(32) or "," & Chr(32) or whatever would be most appropriate.
To populate the lists with existing child data we simply need to "update" one child record for each parent, like so
UPDATE Children SET ChildName=ChildName
WHERE ChildID IN (SELECT MIN(ChildID) AS m FROM Children GROUP BY ParentID)