I have job in Talend that is designed to bring together some data from different databases: one is a MySQL database and the other a MSSQL database.
What I want to do is match a selection of loan numbers from the MySQL database (about 82,000 loan numbers) to the corresponding information we have housed in the MSSQL database.
However, the tables in MSSQL to which I am joining the data from MySQL are much larger (~ 2 million rows), are quite wide, and thus cost much more time to query. Ideally I could perform an inner join between the two tables based on the loan number, but since they are in different databases this is not possible. The inner join that is performed inside a tMap occurs after the Lookup input has already returned its data set, which is quite large (especially since this particular MSSQL query will execute a user-defined function for each loan number).
Is there any way to create a global variable out of the output from the MySQL query (namely, the loan numbers selected by the MySQL query) and use that global variable as an IN clause in the MSSQL query?
This should be possible. I'm not working in MySQL but I have something roughly equivalent here that I think you should be able to adapt to your needs.
I've never actually answered a Stackoverflow question and while I was typing this the page started telling me I need at least 10 reputation to post more than 2 pictures/links here and I think I need 4 pics, so I'm just going to write it out in words here and post the whole thing complete with illustrations on my blog in case you need more info (quite likely, I should think!)
As you can see, I've got some data coming out of the table and getting filtered by tFilterRow_1 to only show the rows I'm interested in.
The next step is to limit it to just the field I want to use in the variable. I've used tMap_3 rather than a tFilterColumns because the field I'm using is a string and I wanted to be able to concatenate single quotes around it but if you're using an integer you might not need to do that. And of course if you have a lot of repetition you might also want to get a tUniqueRows in there as well to save a lot of unnecessary repetition
The next step is the one that does the magic. I've got a list like this:
'A1'
'A2'
'B1'
'B2'
etc, and I want to turn it into 'A1','A2','B1','B2' so I can slot it into my where clause. For this, I've used tAggregateRow_1, selecting "list" as the aggregate function to use.
Next up, we want to take this list and put it into a context variable (I've already created the context variable in the metadata - you know how to do that, right?). Use another tMap component, feeding into a tContextLoad widget. tContextLoad always has two columns in its schema, so map the output of the tAggregateRows to the "value" column and enter the name of the variable in the "key". In this example, my context variable is called MyList
Now your list is loaded as a text string and stored in the context variable ready for retrieval. So open up a new input and embed the variable in the sql code like this
"SELECT distinct MY_COLUMN
from MY_SECOND_TABLE where the_selected_row in ("+
context.MyList+")"
It should be as easy as that, and when I whipped it up it worked first time, but let me know if you have any trouble and I'll see what I can do.
Related
I have a table that contains data from repeated experiments (for example, site A has one sample, and the lab processed the sample three times obtaining slightly different values). I need to average these results in a separate table, but what I have read on the Microsoft support site is that a query that pulls data into another table with a calculated field is not possible on Access.
Can I query multiple data points from one table into a single calculated field in another table? Thank you.
UPDATE
I ended up doing a lot of manual adjustments of the file format to create a calculated field in the existing table that averages each sites data, so my problem is, for my current purposes, solved. However I would still like to understand. Following up with you both, I think the problem was that I had repeated non-unique IDs between rows when I probably should have made data columns with unique variable names so that I could query each variable name for an average.
So, instead of putting each site separately on the y axis, I formatted it by putting the sample number for each site on the x-axis:
I was able to at least create a calculated field using this second format in order to create an average value for each site.
Would have there been a way to write a query using the first method? Luckily, my data set was not at all very hefty, so I could handle a reformat manually, but if the case were with thousands of data entries, I couldn't have done that.
Also, here is the link to the site I mentioned originally https://support.office.com/en-ie/article/add-a-calculated-field-to-a-table-14a60733-2580-48c2-b402-6de54fafbde3.
Thanks all.
In my app the user can select multiple filter options. I store this in a DB table.
For example
User 1 can select filters A^B
User 2 can select filters AORC^D
and so forth.
The way it is stored in Db is
user filter_selected
user1 A^B
user2 AORC^D
Now the criteria is no user can have the same filters selected. So if user 3 comes and select A^B or B^A it should throw a error.
I am trying to come up with a smart logic to validate this in javascript.
One approach is go through all the users in the DB (can be many) and sort alphabetically and check if its the same. So in our example A^B and B^A will be the same AB^. This way I can check. Any other better approach may be using mysql command itself ?
you can sort your filter rule based on character and then insert it to do
for example, B^A will convert to AB^ and when you want to check you can sort your filter and then search it
if you want to have an original filter you don't care about the size of your database and more you care about speed you can save original as another column too.if you are care about size of database you can just save the original filter and when you want to search select the rows that have the same length as your filter and then you need to sort alphabetically or you can save index of every filter chars for example when you change A^B to AB^ you can save this filter AB^|021 but this will need to some more space too like original column and I don't suggest this method. also if your filters are always in small length you can don't fetch all record and compare to all. you can just create all possible way of the filter(for example AB^ A^B B^A BA^ ^AB ^BA) but you must be careful because in this method you are creating n! string and this is not good at all, just for too small length string its ok and that's when you have too many records in your database this method can be good
I have looked and haven't found a method on here to do this. I am assuming my search is skewed and I just missed it, if this is the case, please let me know.
Anywhooo, I have a large and unwieldy report coming out of SAP every day. Because it will often have some strangeness, we import that into an Access database so we can keep an eye on the stuff we need in our department. I am using a combination of 6 fields to create a primary key in Access. The information in those fields is about the only thing consistent I get out of this SAP report, but the remainder of the data can be considered dynamic and can change from day to day. Usually this is a matter of filling in a few blanks, Occasionally this is changing existing data, and on rare occasions, it may involve deleting data out of a handful of fields.
The SAP report is around 130 columns of data, So I'm looking for an efficient way to roll in the changes without overwriting what folks put in there manually.
EDIT:
Here is the way this is used. SAP (for reasons I'm not going to go into) sometimes will have bad data show up in the daily report. We are using Access to track and put the correct data in to something that we can generate much more accurate summaries. What the users put in is to be considered true and accurate.
The transactions we are tracking can take a long time to complete. Most take around 30 days to complete. That's why I will have blank fields on one day, and several of them to be filled in on the next. We might not get any for the next few days and then a bunch more are filled in later. That is the normal flow.
What I have to account for is the odd occasion where a mistake is made early in the process. At a certain point, an error will break SAP's ability to update anything at all in the report we have to use.
I have 3 fields set up that trigger what my users daily work is going to be. There is a logical flow so that user 1 completes what he needs to do and then that record will show up on User 2's report. These fields will also stop the general update process in an exception report if there is a difference in what is coming in from SAP, and what is already in my database.
What I am looking for is some way to systematically fill in blank fields, on existing records in access. I do not want to overwrite if something is in a field, only the null values. I can do this on one field at a time, but each record has about 130 fields. I'm wondering if there is a way I could do this in just 1 query?
Thanks all! I hope the edit makes more sense now
A simple google for "Access SQL update null values" could have yeilded you what you need. But if all you need to do is fill constant values into empty fields then something like:
UPDATE Table SET Table.field1 = VALUE
WHERE Table.field2 is NULL;
Now if this data is different for each record based on; say data from another field, then you may need to write some VBA to build that value/string for you. But otherwise if you are JUST updating null fields to include data, then a simple UPDATE statement will do
EDIT Based on new info:
So if I'm understanding correctly: you have two tables. One table with the blank fields and another table that contains the values you need.
If this is the case, you can use a similar UPDATE statement, but use an inner join to get the data you need from table B to fill in table A
UPDATE TableA INNER JOIN TableB ON TableA.KeyField = TableB.KeyField
SET TableA.NullField = TableB.NotNullField
WHERE TableA.NullField Is NULL;
I am trying to determine the best method of collecting a large list from a database and then displaying and filtering the results on the client side. Let me give a quick example:
Example: I've got a database with customer data and currently it contains around 2000 records. This number is constantly increasing. On my website I have a page that I want to be able to query said database based on information such as name, email, phone number etc. and of course display the results (when a user types in Smith it returns all records containing the name Smith). I am planning on using AJAX so that I can query the database and display the results on the fly similar to how google does it. When a user begins searching, results will start showing up on the page as they are found.
Possible Solutions:
Unfortunately I am stumped on how to go about implementing something like this. I am considering using a ValueList pattern. When the user first loads the page, should I be querying the database and storing every record in a collection and then searching that collection list and displaying the results on my jsp page? Essentially creating a java database. The thing I like about the ValueList pattern is that I take one huge hit on page load and dump the entire database in objects stored in a list. What if the database is larger though, say 2,000,000 records?
Or should I be using a simple DOA pattern without the ValueList and query the database for each individual search? This would result in a LOT of database queries, especially considering that I plan on returning results as the user types in the search box.
Edit: The more I think about this, the more it is an AJAX question. My biggest concern should be how to query my database while the user is typing. Do I set some sort of listener to listen for the user to stop typing and then perform the query?
I would use Solr for this type of task.
Fields, which you are going to use for searching should be indexed with Solr.
Then you do an ajax query to Solr and get the result. You can set the order, number of items per page and show results only for current page.
Solr has a lot of other features that can be useful for you.
I'm creating a form in CodeIgniter that has a paired value- Companies, and CEOs. The idea is, while there is only one Company field with one CEO field below initially on the form, there is a button for me to create more pairs of Company-CEO fields. I have an idea for how to make that appear on the frontend, but I'm not quite sure which is the best way to store it on the backend.
I think in order to combine each Company and CEO field into one value for storing in the MySQL database, I could simply use CONCAT. However, I'm not sure if I should even bother creating such a value. In any case, how would I handle the fact that there are a variable number of Companies (and thus CEOs) for each record? Would I have to store Company and CEO within another value? How could such an array or tuple be handled in MySQL?
Point 1: Do not combine the values. It will give you headaches in code splitting and combining. Make two columns, one for company, one for CEO.
Point 2: In general, when using a database, any time you have a list it gets its own table.
The various JS libraries out there let you handle this with a grid.