I have an a MySQL database. My database contains documents with a datetime column called "created". I want to group by day in order to have the document count per day. However, some days have zero documents and as a result they are not part of the output. For example I need '2001-01-01' to have a zero count if documents do not exist.
I am thinking of creating an extra table with the date range I am interested on and the to Do an outer join with my table. Then I can group by date to have my results.
Is there any better way of doing such a thing?
My SQL code:
Select date(created_at),c.text from Dates d left outer join classifier c on d.n=DATE(c.created_at)
where c.classifier="2014streamlrall"
and date(c.created_at)>='2014-03-01' and date(c.created_at)<='2014-05-01'
order by d.n;
The left join still does not work.
There is no better way for that in MySQL.
It lacks both a method to generate an arbitrary length resultset (similar to PostgreSQL's generate_series) and recursive SQL required to emulate such a method (which is used in SQL Server and Oracle).
Even on SQL Server, populating and keeping a table with 100 years worth of dates (which takes but a little more than 73K records) gives much better performance on reports similar to yours than using a generated resultset.
Is your real issue with having a create a row with '2001-01-01: 0'? If so, just wrap the MySQL with PHP and control your output with PHP. Then you just echo (or create an XML entry, whatever formatting you need) with the date and the result, even if it's 0. Even if you're currently know nothing about PHP, there are plenty of tutorials on how to run MySQL queries in PHP and output results. Good luck.
Related
My customer is using MSAccess to read SQL Server Data.
Originally they created a linked table to the SQLServer base table,
then in Access, created a Query that aggregated and filtered.
Select f1,f2,sum(f3),sum(f4)
From linkedtablename
where fx = 'somevalue'
group by f1,f2
For security & performance reasons, I built simple a query in MSSQL to do the filtering & aggregating,
and asked the user to point to that instead, with a passthrough query.
So now they have and ODBC 'passthroughquery' doing a 'select * from MSSQLview'
However, MSAccess seems to be really struggling when we do anything with this passthrough.
e.g. Adding the passthrough to a new MSAccess query window takes forever.
Seems as if Access is doing some heavy reading of the source or source metadata each time we interact with it.
Running a select against the passthrough is also taking an age ... but with the aggregating being done by MSSQL, it should be a lot faster !?
So the question is, why does MSAccess struggle so much ?
Is Access trying to profile the source data even without an explicit 'select' being done ?
Or is it trying to read metadata every time we interact with the Passthrough ?
Ultimately, I am hoping that there is some configuration setting that will force Access to treat this like a 'standard' table !
If you use a PT query, keep in mind that if you use this as a source for a client query, then ZERO additional filtering can be used, or will work.
In other words?
A PT query is one of the fastest (high performance) ways to pull data. BUT ONLY if you do NOT attempt to add additional filtering. A PT query cannot be filtered by client side (well, it can, but ONLY after pulling the PT full query source).
As a result?
Use a linked view. They perform JUST as well as a stored procedure and a PT query, but they can and do respect client side filtering. So, for example you can build a client side query against that linked view with criteria, and ONLY the records matching that query are pulled down the network pipe.
this seems somewhat counter intuitive, but PT queries are fast, but they DO NOT respect additional filters client side (to be specific, you can filter against a pt query, but Access ONLY does so after pulling all records in the PT query). So, one would do VERY well to say avoid using a PT query to fill a combo box, or any thing else client side that will and does attempt to apply additional filtering and criteria.
To be crystal clear:
A PT query is great, but ONLY if you going to have the PT query do the filter in the first place. Additional filtering can be done, but that assumes the original PT query did not pull a lot of data in the first place. So a PT query rows pulled is what you WILL GET client side.
In 99% of cases, you are FAR better off to put that query in a linked view, and thus you are free to filter and add criteria to that view (even client side), and ONLY records meeting that criteria are pulled down the network pipe. this includes even using a client side query on that linked view. And this also includes basing a report on that linked view, and say you have VBA to add/provide a "where" clause to that report. (in this case, once again, Access will ONLY pull records based on that criteria. If you use a PT query for the report and attempt to filter - that filtering ONLY occurs AFTER all PT records been pulled.
So, PT queries cannot effective REDUCE bandwidth requirements to ANY lower then what the PT query returns in the first place. However, linked views DO allow and DO respect additional filters applied - even when done client side. As a result, a PT query is not all that useful unless the PT query has the criteria pre-defined and known ahead of time.
So, I would strong suggest you try/test/use a linked view.
In other words, put that sum and group by in a server side view and link to that.
Edit: and you CAN add the where clause to that view client side against that view.
However, because Fx is not a column, then you have to either add that column to the view, or create a stored procedure, and use it this way:
CREATE PROCEDURE [dbo].[GetMyGroupSum]
#fx nvarchar(50)
AS
BEGIN
SET NOCOUNT ON;
SELECT f1, f2, SUM(f3), SUM(f4)
FROM dbo.TheTableToQueryOn
WHERE fx = #fx
GROUP BY f1,f2
END
Now, create a PT query in Access. You code will then look like:
dim strFX as string
strFX = "somevalue"
currentdb.querydefs("MyPtQuery").sql = "EXEC GetMyGroupSum '" & strFX & "'"
You are now free to use the above query for a report, code or even launch a form based on that PT query.
In MOST cases you are better to use a view, but because the query does NOT return the column you need to filter on, then a PT query is the solution, but in most cases, it is not.
I'm trying to apply 'curdate()' functionality to a select statement from DB2. I'm used to MySQL but I'm still trying to get the hang of a lot of the DB2 functionality and how to essentially marry the two.
My query is complete except for one line. I'm trying to select based on a ship date, which is the column EXTD1H and I need to check it against today or curdate(). The problem is that column in DB2 is an integer format, not a date format, and I don't have the option of changing it. In prior inserts to mysql, I've been able to put it into Y-m-d format and I know I can trim the year using LEFT(EXTD1H, 4) but I have no idea how to modify my select so that I can say WHERE EXTD1H is today so that I'm only selecting records for this date.
Here's the query:
select
invnoz as ORDER,
fstatz as STATUS
from gportafl
/*where EXTD1H is curdate, hypothetically*/
AND FSTATZ <> 'S'
limit 20;
As you can see, I have a commented line where my issue is. I'm sure it's simple I just can't seem to find in the documentation exactly what I'm looking for, which is to be able to use that INT column to verify that selected records are from today.
UPDATE:
All values from the column are in YYYYMMDD format i.e.
20180202
but it should be 2018-02-02
It's best not to do operations on the columns, so the indexes are used.
You can typecast the current date to fit your data as follows:
WHERE extd1h = INTEGER(VARCHAR_FORMAT(CURRENT DATE,'YYYYMMDD'))
I have job in Talend that is designed to bring together some data from different databases: one is a MySQL database and the other a MSSQL database.
What I want to do is match a selection of loan numbers from the MySQL database (about 82,000 loan numbers) to the corresponding information we have housed in the MSSQL database.
However, the tables in MSSQL to which I am joining the data from MySQL are much larger (~ 2 million rows), are quite wide, and thus cost much more time to query. Ideally I could perform an inner join between the two tables based on the loan number, but since they are in different databases this is not possible. The inner join that is performed inside a tMap occurs after the Lookup input has already returned its data set, which is quite large (especially since this particular MSSQL query will execute a user-defined function for each loan number).
Is there any way to create a global variable out of the output from the MySQL query (namely, the loan numbers selected by the MySQL query) and use that global variable as an IN clause in the MSSQL query?
This should be possible. I'm not working in MySQL but I have something roughly equivalent here that I think you should be able to adapt to your needs.
I've never actually answered a Stackoverflow question and while I was typing this the page started telling me I need at least 10 reputation to post more than 2 pictures/links here and I think I need 4 pics, so I'm just going to write it out in words here and post the whole thing complete with illustrations on my blog in case you need more info (quite likely, I should think!)
As you can see, I've got some data coming out of the table and getting filtered by tFilterRow_1 to only show the rows I'm interested in.
The next step is to limit it to just the field I want to use in the variable. I've used tMap_3 rather than a tFilterColumns because the field I'm using is a string and I wanted to be able to concatenate single quotes around it but if you're using an integer you might not need to do that. And of course if you have a lot of repetition you might also want to get a tUniqueRows in there as well to save a lot of unnecessary repetition
The next step is the one that does the magic. I've got a list like this:
'A1'
'A2'
'B1'
'B2'
etc, and I want to turn it into 'A1','A2','B1','B2' so I can slot it into my where clause. For this, I've used tAggregateRow_1, selecting "list" as the aggregate function to use.
Next up, we want to take this list and put it into a context variable (I've already created the context variable in the metadata - you know how to do that, right?). Use another tMap component, feeding into a tContextLoad widget. tContextLoad always has two columns in its schema, so map the output of the tAggregateRows to the "value" column and enter the name of the variable in the "key". In this example, my context variable is called MyList
Now your list is loaded as a text string and stored in the context variable ready for retrieval. So open up a new input and embed the variable in the sql code like this
"SELECT distinct MY_COLUMN
from MY_SECOND_TABLE where the_selected_row in ("+
context.MyList+")"
It should be as easy as that, and when I whipped it up it worked first time, but let me know if you have any trouble and I'll see what I can do.
I am trying to apply group active record command in rails rest api, however my database is in MySql.
When I query without group by I get correct data but when I use group on the same query I get strange data collection. I am using group to decrease query time coz in original it takes alot of time to retrieve data from database
Here is my original query
Records.owned_by(User.find_by_email(params[:user].to_s).id).where(device_id: params[:did]).includes(:record_students, :record_employees, :record_admins, :record_others)
but when I use group to increase the efficiency the returned data set is not valid
here is my new query with group
Records.owned_by(User.find_by_email(params[:user].to_s).id).where(device_id: params[:did]).includes(:record_students, :record_employees, :record_admins, :record_others).group("date(created_at)")
any idea what is wrong. Thanks
I have joined 5 tables and done transformation on these tables. Now I got a single table at the end. Now I want to perform sql query on this single table to filter records. But I don't know how to perform simple sql query on this table. I have attached a snap shot which shows the resulting table. How I get this resulting data set as the source? I want to populate my destination after filter out this data.
I am using SSIS 2008.
Click here to see the Table on which I want to perform a simple sql query
SELECT * FROM `first_table`
where `some_column` =
(
SELECT `*`
FROM second_table
WHERE
`some_column2`='something'
LIMIT 1
)
Try this code This will help. You can even use this to connect all those four tables with each other.
From the image you posted, it looks like you have a set of data in the dataflow you're trying to query against. You need to do one of two things at this point. Either you insert the data into a table in the database and use another data flow to query it, or you use use a conditional split (or multicast and conditional splits) to filter the rows down further from there.
Without more detail about what you're actually trying to accomplish, these are the recommendations I can determine.
You could send the rows into a record set destination, but you aren't able to query it like a regular table and you'd need some C#/VB skills to access it to do more than a FOR EACH loop.
Assuming your sql query that you want to run against the resulting table is simple, you can use a script component task. By simple, I mean, if it is of this nature:
SELECT * FROM T WHERE a = 'zz' and b = 'XX' etc.
However, if your query has self joins, then you would be better of dumping the outcome of joining those 5 tables in to a physical table, and go from there.
It appears that query is going to be real straight-forward; in that case using a script component would be helpful.
A separate question: It's advisable to do the sorting at the database level. You are using 5 sort tasks in your solution. Can you please elucidate the reason?