When writing to a Phonograph table sometimes queries exceed the maximum allowed size of 190kB. Is there any way to work around this limit?
The context to that question is that users should define custom filters on a dashboard. Upon storing a filter definition I want to keep track of the data items that are matching the current filter to later being able to identify newly matching data items, e.g. when new data arrives in the system. However, the list of data ids that match the filter is sometimes too big, and hence the error occurs.
Thanks,
Tobias
Can you try forcing the user to use a group of filters.
We had this problem and were told that the intention of allowing a small package size of Phonograph is with the thought process that we base our filters on a set of fields that collectively always result a finite set of rows. For example a given set of geo filters/ set of product dim filters.
Related
In Spotfire I have a dashboard that uses both filtering (only one filtering scheme) and multiple markings to show the resulting data in a table.
I have created a data function which takes a column and outputs the data in the column after the active filtering scheme and markings are applied.
However, this output column is only calculated if I have something marked in every marking.
I want the output column to be calculated no matter how many of the markings are being used. Is there a way to do this?
I was thinking I could use an IronPython script to edit the data function parameters for my input column to only check the boxes for markings that are actively being used. However, I can't find how to access those parameters with IronPython.
Thanks!
I think it would be a combination of visuals being set to OR instead of AND for markings (if you have a set of markings that are being set from others).
Also are all the input parameters set to required parameter perhaps unchecking that option would still run the script. In the r script you may want to replace null values as well.
Not too sure without some example.
I have looked and haven't found a method on here to do this. I am assuming my search is skewed and I just missed it, if this is the case, please let me know.
Anywhooo, I have a large and unwieldy report coming out of SAP every day. Because it will often have some strangeness, we import that into an Access database so we can keep an eye on the stuff we need in our department. I am using a combination of 6 fields to create a primary key in Access. The information in those fields is about the only thing consistent I get out of this SAP report, but the remainder of the data can be considered dynamic and can change from day to day. Usually this is a matter of filling in a few blanks, Occasionally this is changing existing data, and on rare occasions, it may involve deleting data out of a handful of fields.
The SAP report is around 130 columns of data, So I'm looking for an efficient way to roll in the changes without overwriting what folks put in there manually.
EDIT:
Here is the way this is used. SAP (for reasons I'm not going to go into) sometimes will have bad data show up in the daily report. We are using Access to track and put the correct data in to something that we can generate much more accurate summaries. What the users put in is to be considered true and accurate.
The transactions we are tracking can take a long time to complete. Most take around 30 days to complete. That's why I will have blank fields on one day, and several of them to be filled in on the next. We might not get any for the next few days and then a bunch more are filled in later. That is the normal flow.
What I have to account for is the odd occasion where a mistake is made early in the process. At a certain point, an error will break SAP's ability to update anything at all in the report we have to use.
I have 3 fields set up that trigger what my users daily work is going to be. There is a logical flow so that user 1 completes what he needs to do and then that record will show up on User 2's report. These fields will also stop the general update process in an exception report if there is a difference in what is coming in from SAP, and what is already in my database.
What I am looking for is some way to systematically fill in blank fields, on existing records in access. I do not want to overwrite if something is in a field, only the null values. I can do this on one field at a time, but each record has about 130 fields. I'm wondering if there is a way I could do this in just 1 query?
Thanks all! I hope the edit makes more sense now
A simple google for "Access SQL update null values" could have yeilded you what you need. But if all you need to do is fill constant values into empty fields then something like:
UPDATE Table SET Table.field1 = VALUE
WHERE Table.field2 is NULL;
Now if this data is different for each record based on; say data from another field, then you may need to write some VBA to build that value/string for you. But otherwise if you are JUST updating null fields to include data, then a simple UPDATE statement will do
EDIT Based on new info:
So if I'm understanding correctly: you have two tables. One table with the blank fields and another table that contains the values you need.
If this is the case, you can use a similar UPDATE statement, but use an inner join to get the data you need from table B to fill in table A
UPDATE TableA INNER JOIN TableB ON TableA.KeyField = TableB.KeyField
SET TableA.NullField = TableB.NotNullField
WHERE TableA.NullField Is NULL;
Ok, so what is the best practice when it comes down to paginating in mysql. Let me make it more clear, let's say that a given time I have 2000 records and there are more being inserted. And I am displaying 25 at a time, I know I have to use limit to paginate through the records. But what am I supposed to do for the total count of my records? Do I count the records every time users click to request the next 25 records. Please, don't tell me the answer straight up but rather point me in the right direction. Thanks!
The simplest solution would be to just continue working with the result set normally as new records are inserted. Presumably, each page you display will use a query looking something like the following:
SELECT *
FROM yourTable
ORDER BY someCol
LIMIT 25
OFFSET 100
As the user pages back and forth, if new data were to come in it is possible that a page could change from what it was previously. From a logical point of view, this isn't so bad. For example, if you had an alphabetical list of products and a new product appeared, then the user would receive this information in a fairly nice way.
As for counting, your code can allow moving to the next page so long as data is there to support a new page being added. Having new records added might mean more pages required to cover the entire table, but it should not affect your logic used to determine when to stop allowing pages.
If your table has a date or timestamp column representing when a record was added, then you might actually be able to restrict the entire result set to a snapshot in time. In this case, you could prevent new data from entering over a given session.
3 sugggestions
1. Only refreshing the data grid, while clicking the next button via ajax (or) storing the count in session for the search parameters opted .
2. Using memcache which is advanced, can be shared across all the users. Generate a unique key based on the filter parameters and keep the count. So you won't hit the data base. When a new record, gets added then you need to clear the existing memcache key. This requires a memache to be running.
3. Create a indexing and if you hit the db for getting the count alone. There won't be much any impact on performance.
I have job in Talend that is designed to bring together some data from different databases: one is a MySQL database and the other a MSSQL database.
What I want to do is match a selection of loan numbers from the MySQL database (about 82,000 loan numbers) to the corresponding information we have housed in the MSSQL database.
However, the tables in MSSQL to which I am joining the data from MySQL are much larger (~ 2 million rows), are quite wide, and thus cost much more time to query. Ideally I could perform an inner join between the two tables based on the loan number, but since they are in different databases this is not possible. The inner join that is performed inside a tMap occurs after the Lookup input has already returned its data set, which is quite large (especially since this particular MSSQL query will execute a user-defined function for each loan number).
Is there any way to create a global variable out of the output from the MySQL query (namely, the loan numbers selected by the MySQL query) and use that global variable as an IN clause in the MSSQL query?
This should be possible. I'm not working in MySQL but I have something roughly equivalent here that I think you should be able to adapt to your needs.
I've never actually answered a Stackoverflow question and while I was typing this the page started telling me I need at least 10 reputation to post more than 2 pictures/links here and I think I need 4 pics, so I'm just going to write it out in words here and post the whole thing complete with illustrations on my blog in case you need more info (quite likely, I should think!)
As you can see, I've got some data coming out of the table and getting filtered by tFilterRow_1 to only show the rows I'm interested in.
The next step is to limit it to just the field I want to use in the variable. I've used tMap_3 rather than a tFilterColumns because the field I'm using is a string and I wanted to be able to concatenate single quotes around it but if you're using an integer you might not need to do that. And of course if you have a lot of repetition you might also want to get a tUniqueRows in there as well to save a lot of unnecessary repetition
The next step is the one that does the magic. I've got a list like this:
'A1'
'A2'
'B1'
'B2'
etc, and I want to turn it into 'A1','A2','B1','B2' so I can slot it into my where clause. For this, I've used tAggregateRow_1, selecting "list" as the aggregate function to use.
Next up, we want to take this list and put it into a context variable (I've already created the context variable in the metadata - you know how to do that, right?). Use another tMap component, feeding into a tContextLoad widget. tContextLoad always has two columns in its schema, so map the output of the tAggregateRows to the "value" column and enter the name of the variable in the "key". In this example, my context variable is called MyList
Now your list is loaded as a text string and stored in the context variable ready for retrieval. So open up a new input and embed the variable in the sql code like this
"SELECT distinct MY_COLUMN
from MY_SECOND_TABLE where the_selected_row in ("+
context.MyList+")"
It should be as easy as that, and when I whipped it up it worked first time, but let me know if you have any trouble and I'll see what I can do.
My client wants to sort products by drag & drop. The drag & drop part is easy with javascript.
My problem is how do I save and get the sort order?
I'm using .net c# and SQL Server 2008.
When I move a product and drop it in a new position I get the id of the product that's moved, product in front and product behind. With this data I want to update the sort order of products.
I was thinking of adding a field with position, but then I guess I have to update every item when position changes.
In general adding an additional position field is the only thing you can do, to get truly arbitrary ordering.
But you can implement it in several ways. Here are two ways I've implemented myself some time ago.
1. Method: Update all position values, by looping over your items and performing an UPDATE statement for every position.
This is easy to implement, but because of the many updates, it's not good for many items and/or large tables. Especially if you do it via Ajax and perform a complete re-ordering on every change in the list.
2. Method: Do a smart update of only the affected rows.
SELECT all items in the current sort order (The "old list") (Usually fast compared to an UPDATE statement)
Iterate over all items from the "new list" and compare each item to the item from the old list at the same position/index. If the items are the same, don't do anything
If the items are different find that item from the old list, which should actually be at that position and update its position value accordingly (Some lookup data structure might be useful here)
That way you only have to perform minimal database updates, but you'll have more complex code.
Personally I'd go with the first way, until the database updates actually become a performance problem.
We have a sort column but yes we have to re-index all rows as things change. You could mitigate this by assigning sort's in large enough increments to allow some level of movability before you have to do this, such as in 10's or 100's but that's not the best solution and I'd be interested to see what other ideas people have.
If you can capture each move programatically (with up and down buttons for example) then you can just swap the position numbers of the row moving and the row being moved. Make sure that you add new rows at the max position + 1.