I have developed a web api in .Net Core 5 which uses dapper to run a tabled valued function and return the SQL results. These results are then used to fill various select boxes on the front end in VueJS. However, when I began to build out my front end more I realized my JSON arrays could be nested to really help reduce the number of requests I make to the server as my select boxes are dependent. For instance, one select box includes states and then next select box relates to the cities in those states. Adjusting the tabled value function to return a single table was easy by adding a innerjoin between my state table in the database and the cities table. The joining key was a field called STATE_ID. Therefore I just have multiple rows due to multiple cities per state. So now what I am trying to figure out is how to take this result in my web api and my table valued function result without the use of models into a nested json array such that my results are as follows:
[{state: 'Maryland', cities :[{city: 'Baltimore'}, {city: 'Harford County'}]} ,
{state: 'Pennsylvania', cities :[{city: 'York'}, {city: 'Fawn Grove'}]}]
Table valued function result from A2Q00001_StateInfo(USERNUMBER):
| State_ID | State_Name | City_Name |
|---------------------|------------------|---------------------|
| 1 | Maryland | Baltimore |
| 1 | Maryland | Harford County |
| 2 | Pennsylvania | York |
| 2 | Pennsylvania | Fawn Grove |
My controller is as follows:
public ActionResult StateAndCities([FromQuery] String USERNUMBER)
{
//We have parameters here just in case we want to use them
IEnumerable queryResult;
String query = "select * from dbo.A2Q00001_StateInfo(#USERNUMBER);";
using (var connection = new SqlConnection(connectionString))
{
queryResult = connection.Query(query, new { USERNUMBER = USERNUMBER });
}
return Ok(queryResult);
}
All of the tutorials I have seen online use models to create the nested JSON object and return it however I am not sure how to create the nested object using the serialization in the Ok() function in asp.net core. Is this even posssible or do I need to perform operations on the queryResult from the dapper query? Any point in the right direction would be great.
My advice: split this into steps. I'm guessing your A2Q00001_StateInfo UDF here returns a State and City column (edit: I was close, it was State_Name, via the edit), among other things. So first step: let's just read that:
class SomeType
{
public string State_Name { get; set; }
public string City { get; set; }
}
//...
var queryResult = connection.Query<SomeType>(
"select State_Name, City from dbo.A2Q00001_StateInfo(#USERNUMBER);",
new { USERNUMBER }).AsList();
This gets our data from the database into local memory. Note that I filtered out irrelevant columns to reduce overheads.
Now, the next step is to structure that data - it looks like you want to aggregate by state, and create an array of the cities in each; so: let's do that:
var structured =
from grp in queryResult.GroupBy(x => x.State_Name)
select new
{
state = grp.Key,
cities = grp.Select(row => new { city = row.City }).ToArray()
};
This gives us a projection (using anonymous types) that does the restructuring we want. Finally, we need to convert it to JSON; this might be as simple as:
return Ok(structured);
Or you might need to use the Json/JsonResult APIs directly. However, now that the data is structured: any JSON serializer should know what we want to do here.
Note: you probably can rewrite all this into a single expression, but: don't do that; you're not trying to impress the compiler - it won't care either way. Make the code clear and obvious for the next person who is going to need to touch it (which might well be you).
Related
Given there is a dataset of messages, defined by following code:
case class Message(id: Int, value: String)
var messages = Seq(
(0, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}"""),
(1, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}""")
).toDF("id", "value").as[Message]
var schema = new StructType().add("action", StringType).add("timestamp", TimestampType)
var res = messages.select(
from_json(col("value").cast("string"), schema)
)
+------------------------------------+
|jsontostructs(CAST(value AS STRING))|
+------------------------------------+
| [update,2017-10-0...|
| [update,2017-10-0...|
What is the best way to access the schema information in a plain map function. The function itself returns a row which has lost all the Type infos. In order to reach to the values one has to specify the type again e.g
res.head().getStruct(0).getValuesMap[TimestampType](Seq("timestamp"))
=> Map[String,org.apache.spark.sql.types.TimestampType] = Map(timestamp -> 2017-10-06 01:01:19.0)
or
res.head().getStruct(0).getString(0)
=> res20: String = update
Is there some better way to access the raw json data without spark sql aggregation functions?
As a rule of thumb:
To use collection API (map, flatMap, mapPartitions, groupByKey, etc.) use strongly typed API - define record type (case class works the best) which reflects the schema and use Encoders to convert things back and forth:
case class Value(action: String, timestamp: java.sql.Timestamp)
case class ParsedMessage(id: Int, value: Option[Value])
messages.select(
$"id", from_json(col("value").cast("string"), schema).alias("value")
).as[ParsedMessage].map(???)
With Dataset[Row] stay with high level SQL / DataFrame API (select, where, agg, groupBy)
I have a situation in m current ssis project. I have a huge excel with customer data. The 2 columns have identifiers for the customer data. Something like below. Rest of columns have actual data.
COLUMN A | COLUMN B
--------------------
NAME | XYZ
ADDRESS1 | 1 STREET
ADDRESS2 | APT A
ZIP | 12345
The challenge is to Read the values into a variable or to a column. This is required for validations to be performed. After the transformation, I need data in below format.
NAME | ADDRESS1 | ADDRESS2 | ZIP
--------------------------------------
XYZ | 1 STREET | APT A | 12345
I may not be able to use Pivot transformation because this data is read using script component for a dynamic columns. Can anyone please provide me a solution for this ?
Thanks for your time.
Nothing out of the box is going to help you. The challenge you face is that your key-value data has an implicit grouping. That is, every time you see the Name key, the next N rows are associated back to the opening row.
Since you're already reading data in from a Script task, do what makes sense and instead of outputting in a KVP, redefine your output buffer to be Name, Address1, Address2, Zip with appropriate lengths. Then, in your script task, don't actually call the AddRow() method until you've reached the end of the file or you've encountered a Name row.
Use a Script component. The code is given below. Assumption: There are no missing field values. This code will get you started. After that you would need to tweak it according to your special requirements.
public class ScriptMain : UserComponent
{
string name = string.Empty;
string address1 = string.Empty;
string address2 = string.Empty;
string zip = string.Empty;
public override void PreExecute()
{
base.PreExecute();
}
public override void PostExecute()
{
base.PostExecute();
}
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (Row.Key.ToUpper().Equals("NAME"))
{
name = Row.Value;
}
else if (Row.Key.ToUpper().Equals("ADDRESS1"))
{
address1 = Row.Value;
}
else if (Row.Key.ToUpper().Equals("ADDRESS2"))
{
address2 = Row.Value;
}
else if (Row.Key.ToUpper().Equals("ZIP"))
{
zip = Row.Value;
OutputRowBuffer.AddRow();
OutputRowBuffer.Name = name;
OutputRowBuffer.Address1 = address1;
OutputRowBuffer.Address2 = address2;
OutputRowBuffer.Zip = zip;
}
}
}
I've been working on the database for a very basic social networking site: all you can do is register for an account and send/accept friend requests. The server is written in Javascript (NodeJS)
I have a method getUser(username) that accesses the database to get the JSON object representing a particular user given their username, and a method to list the friends of a particular user. From the listFriends(username) function, I want to return an array of the user objects rather than just their usernames and ideally I would like to utilize my own get() function rather than altering the SQL query that listFriends() uses
It might be easier to understand with an example. If I have three tables:
TABLE: UsersName
username (unique) | firstName | lastName |
------------------|-----------|------------|
pjfry | Phillip | Fry |
proff | Professor | Farnsworth |
bender | Bender | Rodriguez |
TABLE: UsersProfile (their profile description)
username (unique) | description |
------------------|------------------------|
pjfry | I went to the future |
proff | I am very old |
bender | Destroy all humans |
TABLE: Friendships (for simplicity assume that if (a,b) is an entry then so is (b,a))
user1 | user2
-----------|---------------
bender | pjfry
pjfry | bender
pjfry | proff
proff | pjfry
And a function to get the user object:
//the callback accepts the user object
function get (username, callback) {
db.query(
'select * from (UsersName n, UsersProfile p) where n.username=p.username and n.username=\'' + username + '\'',
function (rows) {
//for simplicity assume that this call always succeeds
//the user object is altered a bit:
callback({
username: rows[0].username,
name: {
first: rows[0].firstName,
last: rows[0].lastName,
full: rows[0].firstName + ' ' + rows[0].lastName
},
profile: {
description: rows[0].description
}
});
}
}
And here is the function to list the friends of a given user
//callback accepts the array of friends
function listFriends(username, callback) {
db.query(
'select user2 from Friendships where user1=\'' + username + '\'',
function(rows) {
//assume this call always succeeds
callback(rows)
}
)
}
The problem here is that listFriends() will just return the array of usernames rather than user objects. How could I modify the listFriends() function so it returns the user objects by utilizing the get() function?
It could be done by modifying the SQL statement in listFriends() but it would be much cleaner to use the get() method so that if the structure of the user object is ever changed, it only needs to be changed in one place.
Trying to keep things as DRY as possible, something like this should fit your requirements:
// Disclaimer: I have not run tested this code myself
//the callback accepts the user object
function get(username, callback) {
getFriendsWithConditions('n.username=\'' + username + '\'', function(rows){
// returns only a single object to the input callback
callback(rows[0]);
});
}
//callback accepts the array of friends
function listFriends(username, callback) {
getFriendsWithConditions('n.username in (select user2 from Friendships where user1=\'' + username + '\')', callback);
}
function getFriendsWithConditions(whereCondition, callback) {
db.query(
'select * from (UsersName n, UsersProfile p) where n.username=p.username and (' + whereCondition + ')',
function(rows) {
// Using ECMAScript 5 Array.map
callback(rows.map(rowToUserObject));
}
);
}
function rowToUserObject(row)
{
return {
username: row.username,
name: {
first: row.firstName,
last: row.lastName,
full: row.firstName + ' ' + row.lastName
},
profile: {
description: row.description
}
};
}
A few things that came to mind as I was writing this:
UserName and UserProfile feel like they should only be one table, but for all I know this is a simplified version of your actual database, so I left them as is.
I used Array.map as defined in ECMAScript 5 which should be available in Node.
The SQL alias names for each table are hardcoded throughout each function, meaning that you would also need to a) set some sort of constant or come of with a convention to keep all aliases consistent throughout all your functions (which can be a pain to maintain as the project grows) or b) look into using an ORM that can do these things for you (and more!) This will help you avoid sweating the details of how to construct queries, leaving you to worry about more important things, like what data you actually need. I'm not terribly familiar with what's available for NodeJS in terms of ORMs, but the official NodeJS wiki should be a good place start.
Might it be better not to destroy the existing functionality of listFriends()?
Instead, my thought would be to create a new function called getFriends() which calls listFriends() and for each returned username, calls get() for it. The concatenated series of results from each get() would serve as the return for getFriends().
This is a fairly simple solution that does what I want in 2 network calls (rather than 1, like I had originally hoped to do).
I added a function getUserArray(usernames) that takes an array of usernames and returns the objects from that. It does this by generating SQL that looks like: "SELECT ... FROM (UsersName n, UsersProfile p) WHERE n.username=p.username and n.username in (?, ?, ?, ...) order by field(username, ?, ?, ?, ...)" and passes the array of usernames as the values to be escaped and replaced with the ?'s.
The functions getUserArray() and getUser() share a lot in common, so common parts of the code for those can easily be put into common helper functions.
The functionality of listFriends() is augmented as follows: just before it would have returned the array of usernames, it calls getUserArray() to map the usernames to objects.
Simple, and only 2 network calls. And it keeps listFriends() from messing with the structure of the user metadata tables.
I am new to linq to sql
I wrote this function:
public ICollection<ICustomer> GetAll()
{
DataClasses1DataContext context = new DataClasses1DataContext();
var customers = from customer in context.Customers select customer;
return customers.ToList().Cast<ICustomer>().ToList();
}
But it always return list of null values.
The database contain 3 records "filled with data" but this function return 3 nulls.
how to fix that?
It may not be able to cast the results properly, have you made your partial Customer object implement ICustomer? If not, that is the reason.
Also you don't have to bring it to a list twice, or even once for that matter since you aren't returning a list, it might be more appropriate to change your signature to List or IEnumerable depending on your usage.
You can test whether or not the cast is succeeding by doing a simple test.
DataClasses1DataContext context = new DataClasses1DataContext();
var customers = from customer in context.Customers select customer;
int numberOfCustomers = customers.Count();
var myCustomers = customers.Cast<ICustomer>(); //you could also do .OfType<ICustomer>();
int numberOfICustomers = myCustomers.Count();
If numberOfCustomers is 3 and numberOfICustomers is 0 then you know that was the issue.
Your problem is almost certainly at the .Cast() method (confirm this by stepping through your code & ensuring that customers is populated correctly).
Does the Customer object implement the ICustomer interface? It sounds like an obvious thing to check but that would be a likely problem.
I am hoping you can help. I am developing a tiered website using Linq to Sql. I created a new class(or object) in DBML designer called memberState. This object is not an actual table in the database. I have this method in my middle layer:
public override IEnumerable(memberState) GetMembersByState(string #state)
{
using (BulletinWizardDataContext context = DataContext)
{
IEnumerable(memberState) mems = (from m in context.Members
join ma in context.MemberAddresses
on m.UserId equals ma.UserId
join s in context.States
on ma.StateId equals s.StateId
where s.StateName == #state
select new memberState
{
userId = m.UserID,
firstName = m.FirstName,
middleInitial = m.MiddleInitial,
lastName = m.LastName,
createDate = m.CreateDate,
modifyDate = m.ModifyDate
}).ToArray(memberState)();
return mems;
}
}
The tables in my joins (Members, States, and MemberAddresses are actual tables in my Database). I created the object memberStates so I could use it in the query above (notice the Select New memberState. When the data is updated on the web page how do I persist the changes back to the Member Table? My Member Table consists of the following columns: UserId, FirstName, MiddleInitial, LastName, CreateDate, ModifyDate. I am not sure how save the changes back to the database.
Thanks,
If I remember correctly, you can create a view from the different tables (Members, States, and MemberAddresses) and add that to the data context. Then any modifications to data in the view object can be saved, and linq to sql will handle the commit correctly as long as all the relationships are clearly setup/defined in both the database and in the data context.
If you have a Member table, the dbml will most likely contain a Member class. To update a member in the database, you will have to create a new Member object, and the Attach it to the BulletinWizardDataContext.Members collection. Something similar to the following code should the trick (I have not tested the code):
using (BulletinWizardDataContext context = DataContext)
{
Member m = new Member() { UserId = userId };
context.Members.Attach(m);
m.FirstName = firstName;
// Set other properties
context.SubmitChanges();
}
Attach must be called before setting the properties. Also, Linq2Sql has some issues with Attach in the case where the properties of your object are set to default values (i.e. 0 for numeric values, false for booleans, null for string etc.). In this case Attach will not generate the correct SQL.
var m = myContext.Members.Single(m=> m.UserID == myMemState.userID);
m.FirstName = myMemState.firstName;
m.MiddleInitial = myMemState.middleInitial;
...
That would be the quick way. It does an additional roundtrip to the db, but will work well. If that's an issue for you, then do Attach like Jakob suggested. For that you have to have to do some extra steps, like reviewing the configuration for optimistic updates and make sure you have the original fields when doing the attach.