Related
I'm making Data Warehouse. To load the table I'm using CASE WHEN THEN. I can load 100 rows using LIMIT 100 no problems. It is fast.
But to load full table 33 787 rows I receive errors:
Error Code: 2013. Lost connection to MySQL server during query
Error Code: 1205. Lock wait timeout exceeded; try restarting transaction
My DBA suggested to optimise query. Can you please, suggest a solution?
Kind regards,
Anna
INSERT INTO dw_coder_fact
(age,
gender,
country_citizen,
country_live,
city_population,
is_ethnic_minority,
language_at_home,
degree_type_id,
school_major,
marital_status_id,
has_children,
children_number,
employment_status,
employment_field_id,
is_software_dev,
months_programming,
is_underemployed,
income,
survey_date,
date_id,
attended_bootcamp,
bootcamp_fulljob_after,
bootcamp_postsalary,
hours_learning,
bootcamploanyesno,
money_for_learning,
has_student_debt,
student_debt_owe,
has_debt,
has_home_mortgage,
home_mortgage_owe,
financially_supporting,
has_financially_dependents,
job_apply_when,
type_of_company_pref,
job_role_interest,
job_where_pref,
expected_earning)
SELECT s.age,
s.gender,
s.countrycitizen,
s.countrylive,
s.citypopulation,
s.isethnicminority,
s.languageathome,
CASE s.schooldegree
WHEN "associate's degree" THEN 1
WHEN "bachelor's degree" THEN 2
WHEN 'high school diploma or equivalent (GED)' THEN 3
WHEN "master's degree (non-professional)" THEN 4
WHEN 'no high school (secondary school)' THEN 5
WHEN 'Ph.D.' THEN 6
WHEN 'professional degree (MBA, MD, JD, etc.)' THEN 7
WHEN 'some college credit, no degree' THEN 8
WHEN 'some high school' THEN 9
WHEN 'trade, technical. or vocational training' THEN 10
ELSE 11
end AS DegreeTypeDescription,
s.schoolmajor,
CASE s.maritalstatus
WHEN 'divorced' THEN 1
WHEN 'married or domestic partnership' THEN 2
WHEN 'separated' THEN 3
WHEN 'single, never married' THEN 4
WHEN 'widowed' THEN 5
ELSE 6
end AS MaritalStatusDescription,
s.haschildren,
s.childrennumber,
s.employmentstatus,
CASE s.employmentfield
WHEN 'architecture or physical engineering' THEN 1
WHEN 'arts, entertainmant, sports, or media' THEN 2
WHEN 'constraction and extraction' THEN 3
WHEN 'education' THEN 4
WHEN 'farming, fishing, and forestry' THEN 5
WHEN 'finance' THEN 6
WHEN 'food and beverage' THEN 7
WHEN 'health care' THEN 8
WHEN 'law enforcement and fire and rescue' THEN 9
WHEN 'legal' THEN 10
WHEN 'office and administrative support' THEN 11
WHEN 'sales' THEN 12
WHEN 'software development' THEN 13
WHEN 'software development and IT' THEN 14
WHEN 'transportation' THEN 15
ELSE 16
end AS EmploymentFieldDescription,
s.issoftwaredev,
s.monthsprogramming,
s.isunderemployed,
s.income,
s.part1starttime,
t.date_id,
s.attendedbootcamp,
s.bootcampfulljobafter,
s.bootcamppostsalary,
s.hourslearning,
s.bootcamploanyesno,
s.moneyforlearning,
s.hasstudentdebt,
s.studentdebtowe,
s.hasdebt,
s.hashomemortgage,
s.homemortgageowe,
s.financiallysupporting,
s.hasfinancialdependents,
s.jobapplywhen,
s.jobpref,
s.jobroleinterest,
s.jobwherepref,
s.expectedearning
FROM new_coders_survey_data s,
dw_time_dim t;
I have a table x with the below fields,
n_i int(10) NOT NULL,
n_m longtext,
n_t varchar(255) DEFAULT NULL,
n_s varchar(50) DEFAULT NULL,
n_c int(10) DEFAULT NULL,
n_d datetime DEFAULT NULL
Data:
1, ABC has kept Hold rating on John Inc for target of $ 125 over a 12 month horizon from the current market price of $ 101., Hold John Inc for target of $ 125: ABC, ABC PCG, 1, 1/27/2006 22:55
2, RB Research has kept Buy rating on Johnson (New York) Inc for target of $ 80 over a 12 month horizon from the current market price of $ 64., Buy Johnson (New York) Inc for target of $ 80: RB Research, RB Research, 1, 1/27/2006 23:03
3, XYZ Research has kept Buy rating on John & John Manufacturing (USA) Inc, a subsidiary of John & John Inc for target of $ 340 from the current market price of $ 270., Buy John & John Manufacturing (USA) Inc for target of Rs.340: XYZ Research, XYZ Research, 1, 1/27/2006 23:06
4, ABCDE Research has upgraded Johnson (New York) Inc to Buy with a target of $ 1000 from the current market price of $ 750., Buy Johnson (New York) Inc for target of $ 1000: ABCDE Research, ABCDE Research, 1, 1/27/2006 23:10
5, JKL Private Client Research has kept Buy rating on John2 Inc, parent company of John & John Manufacturing (USA) Inc with a target price of $ 295 from the current market price of $ 276., Buy John2 Inc for target of $ 295: JKL Private Client Research, JKL Private CLient Research, 1, 1/27/2006 23:12
I would like to create the below table y with data from above table plus a new field n_sy. The search criteria is a string say, BSSN which will be taken from table z and searched in fields n_t and n_m of table x and if found field n_sy in table y would be updated with corresponding field data "BSIS" from table z. Also, the field "n_sy" should be able to hold more than one value. :-(
n_i int(10) NOT NULL,
n_m longtext,
n_t varchar(255) DEFAULT NULL,
n_s varchar(50) DEFAULT NULL,
n_c int(10) DEFAULT NULL,
n_d datetime DEFAULT NULL,
n_sy text NOT NULL
Data:
1, ABC has kept Hold rating on John Inc for target of $ 125 over a 12 month horizon from the current market price of $ 101., Hold John Inc for target of $ 125: ABC, ABC PCG, 1, 1/27/2006 22:55, ABCD12345J
2, RB Research has kept Buy rating on Johnson (New York) Inc for target of $ 80 over a 12 month horizon from the current market price of $ 64., Buy Johnson (New York) Inc for target of $ 80: RB Research, RB Research, 1, 1/27/2006 23:03, ABCD34567L
3, XYZ Research has kept Buy rating on John & John Manufacturing (USA) Inc, a subsidiary of John & John Inc for target of $ 340 from the current market price of $ 270., Buy John & John Manufacturing (USA) Inc for target of Rs.340: XYZ Research, XYZ Research, 1, 1/27/2006 23:06, "ABCD56789A, ABCD45678M"
4, ABCDE Research has upgraded Johnson (New York) Inc to Buy with a target of $ 1000 from the current market price of $ 750., Buy Johnson (New York) Inc for target of $ 1000: ABCDE Research, ABCDE Research, 1, 1/27/2006 23:10, ABCD34567L
5, JKL Private Client Research has kept Buy rating on John2 Inc, parent company of Johnson (New York) Inc with a target price of $ 295 from the current market price of $ 276., Buy John2 Inc for target of $ 295: JKL Private Client Research, JKL Private CLient Research, 1, 1/27/2006 23:12, "ABCD23456K, ABCD34567L"
Table z
`BSCe` double DEFAULT NULL,
`BSSI` text,
`BSSN` text,
`BSSt` text,
`BSGr` text,
`BSFV` int(11) DEFAULT NULL,
`BSIS` text,
`BSIn` text,
`BSInst` text,
`NSSy` text,
`NSSc` text,
`NSSer` text,
`NSDOL` text,
`NSPUV` int(4) DEFAULT NULL,
`NSMktL` int(3) DEFAULT NULL,
`NSIS` text,
`NSEFV` int(11) DEFAULT NULL
12345, ABCD, John Inc, A, B, 10, ABCD12345J, Mine, E, J12345, John, T, 10-Oct-2019, 10, 1, ABCD12345J, 10
12346, XYZ, John2 Inc, A, B, 10, ABCD23456K, Iron, E, J12346, John2, T, 11-Jan-2020, 10, 1, ABCD23456K, 10
12347, JKL, Johnson (New York) Inc, A, B, 10, ABCD34567L, Electricty, E, J12347, John3, T, 7-Dec-2019, 10, 1, ABCD34567L, 10
12348, IJK, John & John Inc, A, B, 10, ABCD45678M, Mine, E, J12348, John & John, T, 19-Apr-2019, 10, 1, ABCD45678M, 10
12349, EFGH, John & John Manufacturing (USA) Inc, A, B, 10, ABCD56789A, IT, E, J12349, John & John Manufacturing Inc, T, 29-May-2019, 10, 1, ABCD56789A, 10
Looking for your help, as the MYSQL table is fairly large.
Use below query,
Creates the table with structure of x
create table y as select * from table x;
Adds column column n_sy
alter table y add n_sy text NOT NULL;
Updates BSSN from z to n_sy in y
update y
inner join z
on (z.BSSN = y.n_t)
set y.n_sy = z.BSIS;
Updates BSSN from z to n_sy in y to hold more than one value
update y
inner join z
on (z.BSSN = y.n_m)
set y.n_sy = y.n_sy || ',' || z.BSIS;
Need customized JSON output--
(I have two files - text file and schema file)
abc.txt -
100002030,Tom,peter,eng,block 3, lane 5,california,10021
100003031,Tom,john,doc,block 2, lane 2,california,10021
100004032,Tom,jim,eng,block 1, lane 1,california,10021
100005033,Tom,trek,doc,block 2, lane 2,california,10021
100006034,Tom,peter,eng,block 6, lane 6,california,10021
abc_schema.txt (field name and position)
rollno 1
firstname 2
lastname 3
qualification 4
address1 5
address2 6
city 7
Zipcode 8
Rules-
First 6 characters of rollno
Need to club address1 | address2 | city
Prefix Address to above
Expected Output-
{"rollno":"100002","firstname":"Tom","lastname:"peter","qualification":"eng","Address":"block 3 lane 5 california","zipcode":"10021"}
{"rollno":"100002","firstname":"Tom","lastname:"john","qualification":"doc","Address":"block 2 lane 2 california","zipcode":"10021"}
{"rollno":"100004","firstname":"Tom","lastname:"jim","qualification":"eng","Address":"block 1 lane 1 california","zipcode":"10021"}
{"rollno":"100005","firstname":"Tom","lastname:"trek","qualification":"doc","Address":"block 2 lane 2 california","zipcode":"10021"}
{"rollno":"100006","firstname":"Tom","lastname:"peter","qualification":"eng","Address":"block 6 lane 6 california","zipcode":"10021"}
I do not wish to hardcode the fields but read from the schema file, the idea is to have reusable code. Something like looping schema file and the text file
A = load 'abc.txt' using PigStorage(',') as (rollno, Fname,Lname,qua,add1,add2,city,Zipcode);
B = foreach A generate rollno, Fname,Lname,qua,concate (add1,add2,city) ,Zipcode;
C= STORE B
INTO 'first_table.json'
USING JsonStorage();
Hope this helps.
I write this code in scala.html to show the MySQL data in an HTML table:
#for(a <- 1 to rowNum){
#rs.next()
<tr>
<td>#rs.getString("building_id")</td>
<td>#rs.getString("building_name")</td>
<td>#rs.getString("building_type")</td>
<td>#rs.getString("address")</td>
</tr>
It gives the following result:
true true true true true true true true true true
id name type address
1 The Floravale condo Westwood Avenue
2 building2 condo Jurong West Street 21
3 building3 hdb Jurong West Street 31
4 building4 hdb Jurong West Street 81
5 building5 hdb Jurong West Street 61
6 building6 hdb Jurong West Street 81
7 building7 hdb Kang Ching Road
8 building8 hdb Kang Ching Road
9 building9 hdb Boon Lay Drive
10 building10 hdb Boon Lay Place
How can I hide the true as the output of #rs.next()?
Or are there other ways to display data? Thanks!
As commented, it is probably that true is being returned, and therefore appearing as a result. A more typical usage would be to collect your rows in a List passed to the view, often in it's own case class (say Building), then use a map:
#buildings.map { building =>
<td>building.id</td>
<td>building.name</td>
...
}
In the below relational table, I want to search all user id's whom are are under any particular sponsorid in hierarchical manner
User Id Name Sponsor Id
10030080 TULSI RAM 10029866
10030079 SARABJEET KAUR 10026043
10030078 MANJIT KAUR 100042580
10030077 GURDAS SINGH 10029579
10030076 Tarlochan Singh 10030072
10030075 Gamer Singh 10029943
10030074 Juginder singh 10030068
10030073 Malkin Singh 10029740
10030072 ParmodKumar 10030018
10030071 Samitra devi 10030070
10030070 Lohan LaL 10030016
10030069 Kashmir Singh 10030057
10030068 Sukhpal singh 10029862
10030067 sandeepSingh 10030066
10030066 GurmelSingh 10030065
10030065 kuldeep kaur 10030064
10030064 harpreet kaur 10030063
10030063 kalwinder kaur 10030062
please suggest the query