SSIS Data Transformation - Rows to Columns - ssis

I have a situation in m current ssis project. I have a huge excel with customer data. The 2 columns have identifiers for the customer data. Something like below. Rest of columns have actual data.
COLUMN A | COLUMN B
--------------------
NAME | XYZ
ADDRESS1 | 1 STREET
ADDRESS2 | APT A
ZIP | 12345
The challenge is to Read the values into a variable or to a column. This is required for validations to be performed. After the transformation, I need data in below format.
NAME | ADDRESS1 | ADDRESS2 | ZIP
--------------------------------------
XYZ | 1 STREET | APT A | 12345
I may not be able to use Pivot transformation because this data is read using script component for a dynamic columns. Can anyone please provide me a solution for this ?
Thanks for your time.

Nothing out of the box is going to help you. The challenge you face is that your key-value data has an implicit grouping. That is, every time you see the Name key, the next N rows are associated back to the opening row.
Since you're already reading data in from a Script task, do what makes sense and instead of outputting in a KVP, redefine your output buffer to be Name, Address1, Address2, Zip with appropriate lengths. Then, in your script task, don't actually call the AddRow() method until you've reached the end of the file or you've encountered a Name row.

Use a Script component. The code is given below. Assumption: There are no missing field values. This code will get you started. After that you would need to tweak it according to your special requirements.
public class ScriptMain : UserComponent
{
string name = string.Empty;
string address1 = string.Empty;
string address2 = string.Empty;
string zip = string.Empty;
public override void PreExecute()
{
base.PreExecute();
}
public override void PostExecute()
{
base.PostExecute();
}
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (Row.Key.ToUpper().Equals("NAME"))
{
name = Row.Value;
}
else if (Row.Key.ToUpper().Equals("ADDRESS1"))
{
address1 = Row.Value;
}
else if (Row.Key.ToUpper().Equals("ADDRESS2"))
{
address2 = Row.Value;
}
else if (Row.Key.ToUpper().Equals("ZIP"))
{
zip = Row.Value;
OutputRowBuffer.AddRow();
OutputRowBuffer.Name = name;
OutputRowBuffer.Address1 = address1;
OutputRowBuffer.Address2 = address2;
OutputRowBuffer.Zip = zip;
}
}
}

Related

Convert Dapper raw SQL Result to Nested JSON Array

I have developed a web api in .Net Core 5 which uses dapper to run a tabled valued function and return the SQL results. These results are then used to fill various select boxes on the front end in VueJS. However, when I began to build out my front end more I realized my JSON arrays could be nested to really help reduce the number of requests I make to the server as my select boxes are dependent. For instance, one select box includes states and then next select box relates to the cities in those states. Adjusting the tabled value function to return a single table was easy by adding a innerjoin between my state table in the database and the cities table. The joining key was a field called STATE_ID. Therefore I just have multiple rows due to multiple cities per state. So now what I am trying to figure out is how to take this result in my web api and my table valued function result without the use of models into a nested json array such that my results are as follows:
[{state: 'Maryland', cities :[{city: 'Baltimore'}, {city: 'Harford County'}]} ,
{state: 'Pennsylvania', cities :[{city: 'York'}, {city: 'Fawn Grove'}]}]
Table valued function result from A2Q00001_StateInfo(USERNUMBER):
| State_ID | State_Name | City_Name |
|---------------------|------------------|---------------------|
| 1 | Maryland | Baltimore |
| 1 | Maryland | Harford County |
| 2 | Pennsylvania | York |
| 2 | Pennsylvania | Fawn Grove |
My controller is as follows:
public ActionResult StateAndCities([FromQuery] String USERNUMBER)
{
//We have parameters here just in case we want to use them
IEnumerable queryResult;
String query = "select * from dbo.A2Q00001_StateInfo(#USERNUMBER);";
using (var connection = new SqlConnection(connectionString))
{
queryResult = connection.Query(query, new { USERNUMBER = USERNUMBER });
}
return Ok(queryResult);
}
All of the tutorials I have seen online use models to create the nested JSON object and return it however I am not sure how to create the nested object using the serialization in the Ok() function in asp.net core. Is this even posssible or do I need to perform operations on the queryResult from the dapper query? Any point in the right direction would be great.
My advice: split this into steps. I'm guessing your A2Q00001_StateInfo UDF here returns a State and City column (edit: I was close, it was State_Name, via the edit), among other things. So first step: let's just read that:
class SomeType
{
public string State_Name { get; set; }
public string City { get; set; }
}
//...
var queryResult = connection.Query<SomeType>(
"select State_Name, City from dbo.A2Q00001_StateInfo(#USERNUMBER);",
new { USERNUMBER }).AsList();
This gets our data from the database into local memory. Note that I filtered out irrelevant columns to reduce overheads.
Now, the next step is to structure that data - it looks like you want to aggregate by state, and create an array of the cities in each; so: let's do that:
var structured =
from grp in queryResult.GroupBy(x => x.State_Name)
select new
{
state = grp.Key,
cities = grp.Select(row => new { city = row.City }).ToArray()
};
This gives us a projection (using anonymous types) that does the restructuring we want. Finally, we need to convert it to JSON; this might be as simple as:
return Ok(structured);
Or you might need to use the Json/JsonResult APIs directly. However, now that the data is structured: any JSON serializer should know what we want to do here.
Note: you probably can rewrite all this into a single expression, but: don't do that; you're not trying to impress the compiler - it won't care either way. Make the code clear and obvious for the next person who is going to need to touch it (which might well be you).

Multiple fields have the same columnName Android Room

I have 3 tables ruser, accounts, accountgroup. Each one has a same column called rsuerId.
I created a POJO class with 3 Embedded objects as below.
class GroupChatItem(
#Embedded
val rUserDto: RUserDto,
#Embedded
val account: AccountDto,
#Embedded
val accountGroup: AccountGroupDto
)
Now, i want to make a query that fetches a GroupChatItem with a given rUserId and accountGroupId like the following.
#Query("""
Select ruser.*, accounts.*, accountgroup.*
from ruser
inner join accounts on accounts.rUserId = ruser.rUserId and accounts.active = 1
inner join accountgroup on accountgroup.rUserId = :rUserId and accountGroup.accountGroupId = :accountGroupId
where ruser.rUserId = :rUserId
""")
suspend fun getGroupChatItem(rUserId: Long, accountGroupId: Int): GroupChatItem
Unfortunately i get the following error.
Multiple fields have the same columnName: rUserId. Field names: rUserDto > rUserId, account > rUserId, accountGroup > rUserId.
I have tried to add a prefix to each embedded object but i get also an error. I dont want to retrieve columns one-by-one because there are many of them.
Is there anything that i missed...??
Thank you
Alternatively you can use the prefix attribute of the Embedded anotation:
class GroupChatItem(
#Embedded(prefix = "user_")
val rUserDto: RUserDto,
#Embedded(prefix = "acc_")
val account: AccountDto,
#Embedded(prefix = "accgr_")
val accountGroup: AccountGroupDto
)
and then alias all the columns of each entity in your SQL query.
I think the prefix attribute is s recent update but I am not sure
I don't believe you have any option other than to have/use :-
a) have distinct columns names across the tables that are to be included in joins (then there is no need to prefix the column names),
or
b) to rename the columns using AS when extracting the values along with a prefix when embedding the entity ensuring that the names match.
I believe that a) would be the simpler option as there is a reduction in the chance of inadvertently using the wrong column name.
As I understand it, the column names have to match for Room to be able to know how to be able to copy a value from the underlying result set, which has no indication of what table a value came from to the value in the returned object or objects.
This is an example of the generated code of a similar scenario 3 embedded entities (User, Office and Places) where some of the column names are the same. They each have and id column and User and Places both have a columns named name.
#Override
public UserOfficePlacesCombined getAllUserOfficePlacesCombined() {
final String _sql = "SELECT user.id AS userid, user.name AS username, office.id AS officeid, office.address AS officeaddress, places.id AS placesid, places.name AS placesname FROM User JOIN Office ON User.id = Office.id JOIN Places ON User.id = Places.id";
final RoomSQLiteQuery _statement = RoomSQLiteQuery.acquire(_sql, 0);
__db.assertNotSuspendingTransaction();
final Cursor _cursor = DBUtil.query(__db, _statement, false, null);
try {
final int _cursorIndexOfId = CursorUtil.getColumnIndexOrThrow(_cursor, "userid");
final int _cursorIndexOfName = CursorUtil.getColumnIndexOrThrow(_cursor, "username");
final int _cursorIndexOfId_1 = CursorUtil.getColumnIndexOrThrow(_cursor, "officeid");
final int _cursorIndexOfAddress = CursorUtil.getColumnIndexOrThrow(_cursor, "officeaddress");
final int _cursorIndexOfId_2 = CursorUtil.getColumnIndexOrThrow(_cursor, "placesid");
final int _cursorIndexOfName_1 = CursorUtil.getColumnIndexOrThrow(_cursor, "placesname");
final UserOfficePlacesCombined _result;
if(_cursor.moveToFirst()) {
final User _tmpUser;
if (! (_cursor.isNull(_cursorIndexOfId) && _cursor.isNull(_cursorIndexOfName))) {
final long _tmpId;
_tmpId = _cursor.getLong(_cursorIndexOfId);
final String _tmpName;
_tmpName = _cursor.getString(_cursorIndexOfName);
_tmpUser = new User(_tmpId,_tmpName);
} else {
_tmpUser = null;
}
final Office _tmpOffice;
if (! (_cursor.isNull(_cursorIndexOfId_1) && _cursor.isNull(_cursorIndexOfAddress))) {
final long _tmpId_1;
_tmpId_1 = _cursor.getLong(_cursorIndexOfId_1);
final String _tmpAddress;
_tmpAddress = _cursor.getString(_cursorIndexOfAddress);
_tmpOffice = new Office(_tmpId_1,_tmpAddress);
} else {
_tmpOffice = null;
}
final Places _tmpPlaces;
if (! (_cursor.isNull(_cursorIndexOfId_2) && _cursor.isNull(_cursorIndexOfName_1))) {
final long _tmpId_2;
_tmpId_2 = _cursor.getLong(_cursorIndexOfId_2);
final String _tmpName_1;
_tmpName_1 = _cursor.getString(_cursorIndexOfName_1);
_tmpPlaces = new Places(_tmpId_2,_tmpName_1);
} else {
_tmpPlaces = null;
}
_result = new UserOfficePlacesCombined();
_result.setUser(_tmpUser);
_result.setOffice(_tmpOffice);
_result.setPlaces(_tmpPlaces);
} else {
_result = null;
}
return _result;
} finally {
_cursor.close();
_statement.release();
}
}
The critical lines are the ones like :-
final int _cursorIndexOfId = CursorUtil.getColumnIndexOrThrow(_cursor, "userid")
This is used to search for the column's names in the Cursor (aka result set) and return the offset to the column, the index then being used to get the actual value from the Cursor.
In your scenario the result set will include some like
rUserId rUserId rUserId*
Which one should it use for which? You may know/understand that first is ruser.rUserId, and that the second is account.rUserId and that the third is accountgroup.rUserId but Room, as it stands, will not know when generating the code. So in all 3 instances when getColumnIndex("rUserId") is used, it will return either 0 (the first) it breaks out of the loop, or 2 if it continues rather than breaks out of the loop (I believe it doesn't break out of the loop).

algorithm verifying data from user beween two tables then insert into another table

Greeting I need to get details from users, in those details the user has I have to validate all the User details validate this details with another table and if the date doesn’t match insert on the table but if it does match then don insert anything, this has to be done for all the users, the domains.
User{
String orderNumber
String dealer
Int UserKm
String dateUser
String adviser
Vehicle vehicle
String dateCreated
Date appointmentDate //this date has to be validated with DateNext
appointmentDate from Appointments domain of it doesn’t exit then you can
insert on that table.
}
Appointments{
User user
Date managementDate
Date lasDataApointies
DateNext appointmentDate
Date NextdAteAppointment
Date callDate
String observations
}
def result = User.executeQuery("""select new map(
mmt.id as id, mmt.orderNumber as orderNumber, mmt.dealer.dealer as
dealer, mmt.UserKm as UserKm, mmt.dateUser as dateUser, mmt.adviser as
adviser, mmt.technician as technician, mmt.vehicle.placa as vehicle,
mmt.dateCreated as dateCreated, mmt.currenKm as currenKm) from User as
mmt """)
def result1=result.groupBy{it.vehicle}
List detailsReslt=[]
result1?.each { SlasDataApointing placa, listing ->
def firsT = listing.first()
int firstKM = firsT.UserKm
def lasT = listing.last()
def lasDataApoint = lasT.id
int lastKM = lasT.UserKm
int NextAppointmentKM = lastKM + 5000
int dayBetweenLastAndNext = lastKM - NextAppointmentKM
def tiDur = getDifference(firsT.dateUser,lasT.dateUser)
int dayToInt = tiDur.days
int restar = firstKM - lastKM
int kmPerDay = restar.div(dayToInt)
int nextMaintenaceDays = dayBetweenLastAndNext.div(kmPerDay)
def nextAppointment = lasT.dateUser + nextMaintenaceDays
detailsReslt<<[placa:placa, nextAppointment:
nextAppointment, manageId:lasDataApoint, nextKmUser: NextAppointmentKM]
}
detailsReslt?.each {
Appointments addUserData = new Appointments()
addUserData.User = User.findById(it.manageId)
addUserData.managementDate = null
addUserData.NextdAteAppointment = null
addUserData.observations = null
addUserData.callDate = it.nextAppointment
addUserData.save(flush: true)
}
println "we now have ${detailsReslt}"
}
Based on the details that are not full and looking at the code I can suggest:
no need to do a query to map you can simply query the list of users and check all the properties like user.vehicle. in any case, you need to check each row.
the groupBy{it.vehicle} is not clear but if needed you can do it using createCriteria projections "groupProperty"
Create 2 service method one for iterating all users and one for each user:
validateAppointment(User user){
/* your validation logic */
....
if (validation term){
Appointments addUserData = new Appointments()
...
}
}
validateAppointments(){
List users = User. list()
users.each{User user
validateAppointment(user)
}
}
you can trigger the validateAppointments service from anywhere in the code or create a scheduled job so it will run automatically based on your needs.
if your list of user is big and also for efficiency you can do bulk update - take a look at my post about it: https://medium.com/meni-lubetkin/grails-bulk-updates-4d749f24cba1
I would suggest to create a Custom Validator using a Service, something like this:
class User{
def appointmentService
...
Date appointmentDate
static constraints = {
appointmentDate validator: { val, obj ->
obj.appointmentService.isDateAppointmentValid(obj.appointmentDate)
}
}
}
But keep in mind that validation may run more often than you think. It is triggered by the validate() and save() methods as you’d expect (as explained in the user guide (v3.1.15)). So I'm not sure if this scenario is the best way to validate àppointmentDate` in your domain, so you have to be careful about that.
Hope this help.

How to calculate columns when loading data into ActivePivot

In the CSV file that I use for loading ActivePivot, I have 2 fields that need to be multiplied together to compute my record's value: price * quantity.
I am using the CSV source with topics and channels. Where can I perform this computation?
you should override the compute method of the ColumnParser, see below. In the following example we get the QuantitySold and the SellingPricePerUnit and we add the result in the Sales column, do not forget to add the Sales column in your store definition:
#Bean
#DependsOn(value="csvSource")
public CSVMessageChannelFactory csvChannelFactory() {
CSVMessageChannelFactory channelFactory = new CSVMessageChannelFactory(csvSource(), datastore);
channelFactory.setCalculatedColumns(ORDERS_TOPIC, DatastoreConfig.ORDERS, Arrays.<IColumnCalculator>asList(
//derive new fields
new ColumnParser("Sales", "double"){
#Override
public Object compute(IColumnCalculationContext context) {
Long qty = (Long) context.getValue("QuantitySold");
Double price = (Double) context.getValue("SellingPricePerUnit");
return (qty == null || price == null) ? null: qty*price;
}
}
));
return channelFactory; }

Linq to SQL foreign key mapping

I'm trying to do a basic lnq to sql foreign key mapping using attributes. This should be really simple but I'm having a hard time finding decent info on it on the net. Where am I going wrong?
Say for example you have a class named User with UserId, FirstName, LastName, Location on it. Location is an object called Location
Location class has LocationId, StreetNum, Name, Suburb
How do I map that with Linq to Sql?
Here's what I'm trying
[Column]
public int LocationId { get; set; }
private EntityRef<Location> _location;
[Required(ErrorMessage = "Please enter your suburb")]
[System.Data.Linq.Mapping.Association(Storage = "_location", ThisKey = "LocationId", IsForeignKey = true)]
public Location Location
{
get { return this._location.Entity; }
set { this._location.Entity = value;
LocationId = value.LocationId;
}
}
I'm getting this error:
The null value cannot be assigned to a member with type System.Double which is a non-nullable value type.
Can anyone help?
Probably, somewhere you've used double datatype (StreetNum, maybe).
In the database corresponding column is marked with NULL flag. Try to use double? or Nullable<double> datatype for your properties.