MySQL 1114 Error, Table /tmp/#sql is full - mysql

I am trying to run TPC-H queries in MySQL server.
Currently, I am using 8.0.23 version (The error did not appear when I used 8.0.19 somehow).
Benchmark size of TPC-H is 10 GB.
I get this error (1114 Error, The table /tmp/#sql~~~ is full) when I submit multiple queries at the same time using 8 threads. There is a chance that identical or similar queries submitted at the same time using different threads.
However, when I just submit a single query in MySQL shell, then it returns a result without the error.
This is the query (there are other queries that raise the same error.)
with year_total as ( select c_customer_id customer_id , c_first_name customer_first_name , c_last_name customer_last_name , c_preferred_cust_flag customer_preferred_cust_flag , c_birth_country customer_birth_country , c_login customer_login , c_email_address customer_email_address , d_year dyear , sum(((ss_ext_list_price-ss_ext_wholesale_cost-ss_ext_discount_amt)+ss_ext_sales_price)/2) year_total , 's' sale_type from tpcds1.customer , tpcds1.store_sales , tpcds1.date_dim where c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk group by c_customer_id , c_first_name , c_last_name , c_preferred_cust_flag , c_birth_country , c_login , c_email_address , d_year union all select c_customer_id customer_id , c_first_name customer_first_name , c_last_name customer_last_name , c_preferred_cust_flag customer_preferred_cust_flag , c_birth_country customer_birth_country , c_login customer_login , c_email_address customer_email_address , d_year dyear , sum((((cs_ext_list_price-cs_ext_wholesale_cost-cs_ext_discount_amt)+cs_ext_sales_price)/2) ) year_total , 'c' sale_type from tpcds1.customer , tpcds1.catalog_sales , tpcds1.date_dim where c_customer_sk = cs_bill_customer_sk and cs_sold_date_sk = d_date_sk group by c_customer_id , c_first_name , c_last_name , c_preferred_cust_flag , c_birth_country , c_login , c_email_address , d_year union all select c_customer_id customer_id , c_first_name customer_first_name , c_last_name customer_last_name , c_preferred_cust_flag customer_preferred_cust_flag , c_birth_country customer_birth_country , c_login customer_login , c_email_address customer_email_address , d_year dyear , sum((((ws_ext_list_price-ws_ext_wholesale_cost-ws_ext_discount_amt)+ws_ext_sales_price)/2) ) year_total , 'w' sale_type from tpcds1.customer , tpcds1.web_sales , tpcds1.date_dim where c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk group by c_customer_id , c_first_name , c_last_name , c_preferred_cust_flag , c_birth_country , c_login , c_email_address , d_year ) select t_s_secyear.customer_id , t_s_secyear.customer_first_name , t_s_secyear.customer_last_name , t_s_secyear.customer_login from year_total t_s_firstyear , year_total t_s_secyear , year_total t_c_firstyear , year_total t_c_secyear , year_total t_w_firstyear , year_total t_w_secyear where t_s_secyear.customer_id = t_s_firstyear.customer_id and t_s_firstyear.customer_id = t_c_secyear.customer_id and t_s_firstyear.customer_id = t_c_firstyear.customer_id and t_s_firstyear.customer_id = t_w_firstyear.customer_id and t_s_firstyear.customer_id = t_w_secyear.customer_id and t_s_firstyear.sale_type = 's' and t_c_firstyear.sale_type = 'c' and t_w_firstyear.sale_type = 'w' and t_s_secyear.sale_type = 's' and t_c_secyear.sale_type = 'c' and t_w_secyear.sale_type = 'w' and t_s_firstyear.dyear = 1999 and t_s_secyear.dyear = 1999+1 and t_c_firstyear.dyear = 1999 and t_c_secyear.dyear = 1999+1 and t_w_firstyear.dyear = 1999 and t_w_secyear.dyear = 1999+1 and t_s_firstyear.year_total > 0 and t_c_firstyear.year_total > 0 and t_w_firstyear.year_total > 0 and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else null end and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end > case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else null end order by t_s_secyear.customer_id , t_s_secyear.customer_first_name , t_s_secyear.customer_last_name , t_s_secyear.customer_login limit 100
I checked this thread,
error while executing sql file (ERROR 1114 mysql table is full)
However, It does not help. Here is the list that I checked from the link above.
I have enough disk space.
I increased the size of max_heap_table_size and tmp_table_size as below.
I restarted MySQL server.
innodb_data_file_path is set with autoextend. So, I expect it to extend automatically.
I also checked that my tmpdir is set properly and there is enough disk space.
Here are my MySQL configurations.
innodb_buffer_pool_size=16GB
select ##innodb_buffer_pool_size; -> 17179869184
select ##max_heap_table_size; -> 2147483648
select ##tmp_table_size; -> 2147483648
+-----------------------+------------------------+
| Variable_name | Value |
+-----------------------+------------------------+
| innodb_data_file_path | ibdata1:10M:autoextend |
+-----------------------+------------------------+
Any suggestions will be appreciated. Thanks for reading.

It turns out, I need to change temptable_max_ram (default is 1 GB) since MySQL uses TempTable as default.
If I set internal_tmp_mem_storage_engine as MEMORY (from TempTable), then changing tmp_table_size, max_heap_table_size is fine.

Related

How to get consolidated output table in single line

I have created sample database
create table items (Order_date date, frame char(2) , group_name varchar(50) , order_qty int(4) , receive_qty int(4));
insert into items(Order_date , frame , group_name , order_qty ,receive_qty) values
("2021-3-10" , "am" , "books" , 4280 , 4000),
("2021-3-12" , "pm" , "notebooks" , 3259 , 3100),
("2021-3-14" , "pm" , "erasers" , 2828 , 2500),
("2021-3-15" , "am" , "books" , 3147 , 3088),
("2021-3-16" , "pm" , "notebooks" , 2897 , 2700),
("2021-3-19" , "am" , "notebooks" , 4793 , 3030),
("2021-3-21" , "pm" , "erasers" , 3317 , 3100);
#Query to get Fill Rate
select group_name , (sum(receive_qty)/sum(order_qty)*100) as Fill_Rate
from items
group by group_name;
#Query to get Fill Rate when frame is AM
select group_name , (sum(receive_qty)/sum(order_qty)*100) as AM_Fill_Rate
from items
where frame = "am"
group by group_name;
#Query to get Fill Rate post 16 mar 2021
select group_name , (sum(receive_qty)/sum(order_qty)*100) as Fill_Rate_post_16Mar
from items
where Order_date >= "2021-03-16"
group by group_name;
How can I get Fill_Rate , AM_Fill_Rate and Fill_Rate_post_16Mar in single table in single run??
since i can get all 3 by individually running above 3 queries , 3 times
Please suggest !
select group_name ,
(sum(receive_qty)/sum(order_qty)*100) as Fill_Rate ,
(sum(case when frame = "am" then receive_qty else 0 end)/sum(case when frame = "am" then order_qty else 0 end)*100) as AM_Fill_Rate
(sum(case when Order_date >= "2021-03-16" then receive_qty else 0 end)/sum(case when Order_date >= "2021-03-16" then order_qty else 0 end)*100) as Fill_Rate_post_16Mar
from items
group by group_name;

Is it possible to pull from multiple tables and then insert the result into another table in the same query with MySQL?

I'm querying multiple tables with UNION ALL and then using the alias to insert the result into another table. It's telling me that I have a syntax error right after the INSERT INTO line. The query looks correct to me and the error is pretty vague. Any help would be appreciated!
SELECT
uploads.campaign_name as 'Campaign Name'
, date as Date
, country as Country
, region as Region
, amount_spent as 'Amount Spent'
, impressions as Impressions
, clicks as Clicks
, cpc as CPC
, ctr as CTR
, cpm as CPM
, category as Category
, subcategory as Subcategory
, source as Source
FROM (SELECT
facebook.ad_name as campaign_name
, facebook.day as 'date'
, CASE WHEN facebook.account_name = 'PMD Beauty' THEN 'US'
WHEN facebook.account_name = 'PMD Beauty United Kingdom' THEN 'UK'
WHEN facebook.account_name = 'PMD Beauty Australia' THEN 'AU'
WHEN facebook.account_name = 'PMD Beauty Europe' THEN 'EU' END as country
, facebook.region as region
, facebook.amount_spent as amount_spent
, facebook.impressions as impressions
, facebook.clicks as clicks
, facebook.cpc as cpc
, facebook.ctr as ctr
, facebook.cpm as cpm
, 'Facebook' as source
FROM
facebook
UNION ALL
SELECT
tiktok.campaign_name as campaign_name
, tiktok.date as 'date'
, tiktok.location as country
, null as region
, tiktok.cost as amount_spent
, tiktok.impression as impressions
, tiktok.click as clicks
, tiktok.cpc as cpc
, tiktok.ctr as ctr
, tiktok.cpm as cpm
, 'Tiktok' as source
FROM
tiktok
UNION ALL
SELECT
pinterest.ad_name as campaign_name
, pinterest.date as 'date'
, trim(substring_index(pinterest.targeting_value, ':', 1)) as country
, trim(substring_index(pinterest.targeting_value, ':', -1)) as region
, pinterest.spend_in_account_currency as amount_spent
, pinterest.impressions as impressions
, pinterest.link_clicks as clicks
, pinterest.cpc as cpc
, pinterest.ctr as ctr
, pinterest.cpm as cpm
, 'Pinterest' as source
FROM
pinterest
UNION ALL
SELECT
steelhouse.campaign_group_name as campaign_name
, steelhouse.date as 'date'
, null as country
, null as region
, steelhouse.total_spend as amount_spent
, steelhouse.impressions as impressions
, null as clicks
, null as cpc
, null as ctr
, null as cpm
, 'SteelHouse' as source
FROM
steelhouse
UNION ALL
SELECT
criteo.campaign as campaign_name
, criteo.day as 'date'
, null as country
, null as region
, criteo.cost as amount_spent
, criteo.displays as impressions
, criteo.clicks as clicks
, criteo.cpc as cpc
, criteo.ctr as ctr
, criteo.cpm as cpm
, 'Criteo' as source
FROM
criteo
UNION ALL
SELECT
hivewyre.creative_name as campaign_name
, hivewyre.date as 'date'
, 'US' as country
, null as region
, hivewyre.spend as amount_spent
, hivewyre.imps as impressions
, hivewyre.clicks as clicks
, null as cpc
, null as ctr
, null as cpm
, 'Hivewyre' as source
FROM
hivewyre
UNION ALL
SELECT
generic_social_media.campaign_name as campaign_name
, generic_social_media.date as 'date'
, generic_social_media.country as country
, generic_social_media.region as region
, generic_social_media.amount_spent as amount_spent
, generic_social_media.impressions as impressions
, generic_social_media.clicks as clicks
, generic_social_media.cpc as cpc
, generic_social_media.ctr as ctr
, generic_social_media.cpm as cpm
, 'Generic Template' as source
FROM
generic_social_media) uploads
LEFT JOIN
product_to_category
ON uploads.campaign_name LIKE BINARY concat('%', product_to_category.product, '%')
WHERE uploads.date >= :startDate
INSERT INTO
marketing (campaign_name, category, subcategory, 'date', country, region, amount_spent, impressions, clicks, cpc, ctr, cpm, source, plenadata__created_at, plenadata__updated_at)
SELECT
campaign_name
, category
, subcategory
, 'date'
, country
, region
, amount_spent
, impressions
, clicks
, cpc
, ctr
, cpm
, source
, NOW()
, NOW()
FROM
uploads
In the terminal:
`error`='\t\t\t\tmarketing (campaign_name, category, subcategory, \'date\', country\' at line 149',`stack`='SequelizeDatabaseError: You have an error in your SQL syntax;

Alternate for Indexing

We need result from Table based on datetime filter. But it working slow in mysql. I can't implement indexing on date & timestamp columns due to it will slow our insertion/updation. So can you please suggest any alternate for select the data quickly based on date and datetime filter with better performance.
SQL Query :
SELECT *
FROM (
SELECT id
, title
, language
, lang_code
, financial
, fname
, lname
, mname
, mname_br
, suffix
, CASE WHEN DOB='0000-00-00' THEN NULL ELSE DOB END AS DOB
, street
, street2
, postal_code
, zip_ext
, city
, state
, country_code
, phone_home
, phone_biz
, phone_biz_ext
, phone_contact
, phone_cell
, status
, CASE WHEN date='0000-00-00 00:00:00' THEN NULL ELSE
CAST(date as datetime) END AS date
, sex
, referrer
, referrerID
, providerID
, ethnoracial
, pid
, temp_key
, primary_care
, default_facility
, created_by
, patientStatus
, primary_care_id
, Sec_HCFA
, noBalanceBill
, erx_entry
, erx_patient_id
, athenaID
, CASE WHEN licenseDate='0000-00-00 00:00:00' THEN NULL ELSE licenseDate end as licenseDate
, race
, otherRace
, ethnicity
, otherEthnicity
, primary_care_phy_name
, primary_care_phy_id
, CASE WHEN dod_patient='0000-00-00' THEN NULL ELSE dod_patient END AS dod_patient--
, locked--
, co_man_phy--
, co_man_phy_id--
, vip--
, External_MRN_1--
, External_MRN_2--
, External_MRN_3--
, External_MRN_4
, as_id
, CASE WHEN acc_statement_date='0000-00-00' THEN acc_statement_date END AS acc_statement_date
, CASE WHEN timestamp='0000-00-00 00:00:00' THEN NULL ELSE timestamp END AS timestamp
, api_id
, fmh_pt_status
, race_code
, ethnicity_code
, patient_payer
, CASE WHEN date='0000-00-00 00:00:00' THEN NULL ELSE date END AS transfer_created
,CASE WHEN timestamp='0000-00-00 00:00:00' THEN NULL ELSE timestamp END AS transfer_updated
,CASE WHEN date > '2020-11-10 00:00:00' THEN 'new' ELSE 'changed' END AS flagfield
,CASE WHEN date='0000-00-00 00:00:00' THEN NULL ELSE date END AS sortdate
FROM patient_data
WHERE (date > '2020-11-10 00:00:00' or timestamp > '2019-04-01 19:53:57-04')
AND month(date) > 0)t
ORDER BY flagfield desc,
sortdate;
)
id Column has indexing in the table
Get rid of
SELECT *
FROM (
)t
it adds nothing, except to slow things down.
Let's focus on
SELECT id
FROM patient_data
WHERE (date > '2020-11-10 00:00:00'
or timestamp > '2019-04-01 19:53:57-04')
AND month(date) > 0
Does that shortened query run "too slow"? If not, then we can use that as a derived table to see if that will speed it up. If so, then we will need to get into UNION and indexing.

Combining Two SQL Select Queries with Where Clauses

I have two Oracle queries that I need combined through an inner join where the tables are joined using the person_uid field. This is because I need to compare what an employee's pay, job title, and supervisor was from one year to the next. I need to have the 2015 data and the 2014 data in the same row for each employee, so if this can be done by doing a subquery using an inner join on the person_uid field, that is the method that I believe will accomplish this.
Here is the first query that pulls the necessary 2015 data:
SELECT person_uid,
id ,
position_contract_type,
position,
job_suffix,
position_status,
effective_date,
position_employee_class,
timesheet_organization ,
appointment_pct ,
annual_salary ,
per_pay_salary ,
hourly_rate ,
position_title ,
academic_title ,
supervisor_id ,
supervisor_name ,
supervisor_position ,
supervisor_job_suffix ,
supervisor_title ,
assignment_grade ,
position_change_reason ,
position_change_reason_desc
FROM employee_position_cunm posn
WHERE posn.position_contract_type = 'P'
AND posn.position_status <> 'T'
AND posn.effective_date = (SELECT MAX(effective_date)
FROM employee_position_cunm p2
WHERE p2.person_uid = posn.person_uid
AND p2.position = posn.position
AND p2.job_suffix = posn.job_suffix
AND p2.effective_date <= '01-Nov-2015')
order by person_uid
I need it to be joined to this query on the person_uid field so that each unique ID for the employee has the records for both years in a single row:
SELECT person_uid,
id ,
position_contract_type,
position,
job_suffix,
position_status,
effective_date,
position_employee_class,
timesheet_organization ,
appointment_pct ,
annual_salary ,
per_pay_salary ,
hourly_rate ,
position_title ,
academic_title ,
supervisor_id ,
supervisor_name ,
supervisor_position ,
supervisor_job_suffix ,
supervisor_title ,
assignment_grade ,
position_change_reason ,
position_change_reason_desc
FROM employee_position_cunm posn
WHERE posn.position_contract_type = 'P'
AND posn.position_status <> 'T'
AND posn.effective_date = (SELECT MAX(effective_date)
FROM employee_position_cunm p2
WHERE p2.person_uid = posn.person_uid
AND p2.position = posn.position
AND p2.job_suffix = posn.job_suffix
AND p2.effective_date <= '01-Nov-2014')
order by person_uid
An easy way would be to use OR:
WHERE posn.position_contract_type = 'P' AND
posn.position_status <> 'T' AND
(posn.effective_date = (SELECT MAX(effective_date)
FROM employee_position_cunm p2
WHERE p2.person_uid = posn.person_uid
p2.position = posn.position AND
p2.job_suffix = posn.job_suffix AND
p2.effective_date <= '01-Nov-2014'
) OR
posn.effective_date = (SELECT MAX(effective_date)
FROM employee_position_cunm p2
WHERE p2.person_uid = posn.person_uid
p2.position = posn.position AND
p2.job_suffix = posn.job_suffix AND
p2.effective_date <= '01-Nov-2015'
)
)
In Oracle you could do a UNION or a UNION ALL.
SELECT person_uid,
id ,
position_contract_type,
position,
job_suffix,
position_status,
effective_date,
position_employee_class,
timesheet_organization ,
appointment_pct ,
annual_salary ,
per_pay_salary ,
hourly_rate ,
position_title ,
academic_title ,
supervisor_id ,
supervisor_name ,
supervisor_position ,
supervisor_job_suffix ,
supervisor_title ,
assignment_grade ,
position_change_reason ,
position_change_reason_desc
FROM employee_position_cunm posn
WHERE ...
...
...
UNION ALL
SELECT person_uid,
id ,
position_contract_type,
position,
job_suffix,
position_status,
effective_date,
position_employee_class,
timesheet_organization ,
appointment_pct ,
annual_salary ,
per_pay_salary ,
hourly_rate ,
position_title ,
academic_title ,
supervisor_id ,
supervisor_name ,
supervisor_position ,
supervisor_job_suffix ,
supervisor_title ,
assignment_grade ,
position_change_reason ,
position_change_reason_desc
FROM employee_position_cunm posn
WHERE ....
....
....;

how to calculate from each row value in sql

I have a table named general_ledger from which I need to show dr_amount, cr_amount and the balance between them as running_balance. That's why I have written a query that is given below. But I am getting the result of each query like the balance only of current row. But I need to produce the result with the remaining balance. Suppose First row dr_balance is 20000 and cr_balance is 5000 and remaining balance is 15000. In second row only cr_balance is 5000. Now the result should be 10000 with the deduction but my result is -5000. I have no idea how to fix this. Can anyone please help me on this? I need your help very much. Here is my query given below :
SELECT '' AS cost_center_id
, '' AS cost_center_name
, '' AS office_code
, CONVERT('2013-02-01',DATETIME) AS transaction_date
, '' AS accounts_head_id
, '' AS account_name
, '' AS opposite_accounts_head_id
, '' AS opposite_account_name
, 'Opening Balance' AS particulars
, tempOpeningBalance.dr_amount
, tempOpeningBalance.cr_amount
, '' AS voucher_no
, '' AS vin
FROM (SELECT IFNULL(mcoa.account_code,'1101010101100321') AS account_code
, IFNULL(mcoa.account_name,'Cash') AS account_name
, IFNULL(mcoa.account_type,'ASSET') AS accountType
, CAST(IFNULL(SUM(IFNULL(maingl.dr_balance,0)),0) AS DECIMAL(27,5)) AS dr_amount
, CAST(IFNULL(SUM(IFNULL(maingl.cr_balance,0)),0) AS DECIMAL(27,5)) AS cr_amount
FROM master_chart_of_accounts AS mcoa
INNER JOIN chart_of_accounts AS coa ON (mcoa.id = coa.master_chart_of_accounts_id AND mcoa.id = 80)
LEFT JOIN general_ledger AS maingl ON (coa.id = maingl.accounts_head_id AND coa.account_code='1101010101100321')
INNER JOIN
( SELECT gl.accounts_head_id, MAX(gl.gl_id) AS max_gl_id, gl.office_code, gl.office_type, gl.country_id,gl.cost_center_id
FROM general_ledger AS gl
-- INNER JOIN voucher_info AS vi ON (gl.voucher_info_id = vi.id)
-- WHERE vi.posting_date < '2013-02-01' AND
WHERE gl.transaction_date < '2013-02-01' AND
gl.cost_center_id IN ('BI0000000000000000000001') AND
gl.country_id IN (1) AND
gl.office_code IN ('UG500013') AND
1=1
GROUP BY gl.accounts_head_id, gl.office_code, gl.office_type, gl.country_id,gl.cost_center_id
ORDER BY gl.accounts_head_id
) AS tmpgl
ON ( maingl.office_code = tmpgl.office_code
AND maingl.office_type = tmpgl.office_type
AND maingl.accounts_head_id = tmpgl.accounts_head_id
AND maingl.country_id = tmpgl.country_id
AND maingl.cost_center_id = tmpgl.cost_center_id
AND maingl.gl_id = tmpgl.max_gl_id
)
WHERE mcoa.account_status_id = 1 AND
coa.account_status_id = 1
) AS tempOpeningBalance
UNION
SELECT vi.cost_center_id
, cc.center_name AS cost_center_name
, gl.office_code
, vi.posting_date AS transaction_date
, vd.accounts_head_id
, (SELECT chart_of_accounts.account_name FROM chart_of_accounts WHERE chart_of_accounts.id = vd.accounts_head_id) AS account_name
, vd.opposite_accounts_head_id
, (SELECT chart_of_accounts.account_name FROM chart_of_accounts WHERE chart_of_accounts.id = vd.opposite_accounts_head_id) AS opposite_account_name
, vd.particulars
, gl.dr_amount AS dr_amount -- here to check
, gl.cr_amount AS cr_amount
, vi.voucher_no
, vi.vin
FROM general_ledger AS gl
INNER JOIN voucher_info AS vi
ON (gl.voucher_info_id = vi.id)
INNER JOIN cost_center AS cc
ON (vi.cost_center_id = cc.id)
INNER JOIN voucher_details AS vd
ON (vi.id = vd.voucher_info_id)
INNER JOIN chart_of_accounts AS coa
ON (vd.accounts_head_id = coa.id)
WHERE vi.posting_date BETWEEN '2013-02-01' AND'2013-02-28'
AND vi.voucher_status_id = 3
AND vd.status_id = 1
AND vi.office_code = 'UG500063'
AND coa.account_code='1101010101100321'
AND coa.cost_center_id = 'BI0000000000000000000001'
ORDER BY cost_center_name
, office_code
, transaction_date;
Use a variable like this
SET #running_balance=0;
SELECT dr_amount AS dr_amount
, cr_amount AS cr_amount
, #running_balance := (#running_balance + dr_amount - cr_amount)
FROM general_ledger