Writing in Linux/Lucee and Coldfusion/Mysql. I have several databases in my system, with quite a few tables overlapping. I have a program which would add a field to a given table, say Activity, in each database. Of course I don't want to alter a non-existent table, so I am checking to make sure the table is there.
The simplest way would be to go to the database and use "show tables like 'Activity'". To do that in ColdFusion I used the datasource that belongs to that database. However, when I tried that, I got zero tables even when the table is present.
<cfquery name = 'tbl' datasource = '#thedatasource#'>
show tables like 'Activity'
</cfquery>
The record count for this query is zero. Then I tried using the information schema and got better results with this test program:
<cfquery name = 'dblist' datasource = 'Moxart'>
select MoxcoId from Moxco
</cfquery>
<cfloop query = 'dblist'>
<cfoutput>
<cfquery name = 'tbl' datasource = 'Moxart'>
select table_schema,table_name from information_schema.tables
where table_schema = '#MoxcoId#' AND table_name = 'Activity'
</cfquery>
<cfdump var = '#tbl#'>
</cfoutput>
</cfloop>
This gave exactly the right answer. However, when I put the same code into the real program, it keeps returning zero records. The real program is rather long, however most of the beginning consists of picking up info from the previous form . From the line
<cfquery name = 'dblist' datasource = 'Moxart'>
the only difference is the addition of this section because #MoxcoId# is not always the schema name. The variable #mmox# is in this case blank, so doesn't add anything.
<cfif #MoxcoId# EQ 'Moxart'>
<cfset pref = '#mmox#Moxware'>
<cfelse>
<cfset pref = '#mmox##MoxcoId#'>
</cfif>
When I do a dump on the query 'tbl' it shows that zero records were retrieved, even though the code appears to be exactly correct.
I am baffled. There must be something wrong in my code, but I just can't see it. If anyone can suggest something I would appreciate it.
Related
I'm a beginner so I Want to know how you can check if a name already exist in the database and give an alert or a message saying the name already exist in the database
but i don't want to use jquery or js just cf and the queries.
<cfif isDefined("Button")>
<cfquery name='Insert' datasource='mysql'>
INSERT INTO tbl_products_manager
(Name)
VALUES ('#name#')
</cfquery>
<cfinclude template="pr.cfm">
</cfif>
I want to know what i should write in the next cfquery or to add the message
You can do:
<cfquery name='Insert' datasource='mysql' result='local.stResult'>
INSERT INTO tbl_products_manager
(Name)
VALUES (<cfqueryparam cfsqltype="CF_SQL_VARCHAR" value="#name#" />)
WHERE NOT EXISTS (SELECT 1 FROM tbl_products_manager WHERE Name = <cfqueryparam cfsqltype="CF_SQL_VARCHAR" value="#name#" />)
</cfquery>
Then you can dump local.stResult and figure out the result value that has the modified record count. (Can't remember offhand.) If the value is 0, you know it wasn't inserted.
As you indicate you are new to Coldfusion, please, please look up <cfqueryparam> and use it in every...single...query, every time.
The above is but one solution. You can also do a separate query first. Or you can have an identity (or MySQL equivalent) on the table and also get that in the stResult struct. There are a myriad of ways. Strenghten your sql-fu, and your CFML query/proc usage will get stronger, too!
I had this code which works fine when the database is small with few records, it writes the json to the file properly, but when the data is huge, it just times out
<cfloop list="t" index="k">
<cfquery name="qry">
select * from #k#
</cfquery>
<cfset js= serializeJSON(qry,'struct')>
<cffile action="write" file="#k#" output="#js#">
</cfloop>
I tried using threads but they are also not working, it just creates empty tables files with no values if i use cfthread with joins
Thought of splitting the files into a combination of 1000 records for each table and and then doing like
table_1, table2, table3, of the same table which is table because it has millions of records and skip for those if they have less than 1000 records to create only 1 file.
but i am just thinking which approach is best and a starting pointing is needed
First of all, let's split this up:
Resultset from database
<cfquery name="qry">
select * from #k#
</cfquery>
Database server retrieves data and streams it via network to the ColdFusion server
ColdFusion stores the data in a query object and stores it in the heap
Serializing the resultset from database
<cfset js= serializeJSON(qry,'struct')>
ColdFusion recursively serializes the whole query object
ColdFusion creates a string object that contains the serialized data and stores it in the heap
Writing the serialized resultset from memory onto the filesystem
<cffile action="write" file="#k#" output="#js#">
ColdFusion writes the string object into a file on the filesystem
Doing all of this within the same request/thread
<cfloop list="t" index="k">
...
</cfloop>
Conclusion
Your code tortures the JVM heap, because references have to be kept until the end of each iteration. The GC can only clean up after a full table has been processed. Large tables (1.000.000+ rows) will likely kill the thread or even hang the JVM.
The Fix: Resultset from database
Retrieving large resultsets at once will always hurt performance. While streaming lots of data within a local network (assuming the database is in the same network) just takes a bit more time, the memory required to store the full resultset is going to be an issue for the JVM.
Instead of doing everything at once, consider splitting it up in smaller chunks of data. Use OFFSET and FETCH in the SQL statement to limit the number of rows per loop. Having multiple iterations will allow the Java GC to free up memory used by previous iterations, relieving the heap.
The Fix: Serializing the resultset from database
Same issue. Big datasets whill hurt performance. Split the resultset by serializing row by row instead of all rows at once.
Writing the serialized resultset from memory onto the filesystem
While this one probably doesn't need a fix, you eventually have to switch to writing line after line.
Some code
<cfset maxRowsPerIteration = 50000>
<cfloop list="t" index="k">
<!--- create empty file to append lines later --->
<cfset fileWrite(k, "")>
<cfset rowsOffset = 0>
<!--- NOTE: you might want to lock the table (prevent write access) here --->
<!--- infinite loop will be terminated as soon the query no longer returns any rows --->
<cfloop condition="true">
<!--- fetch a slice of the full table --->
<cfquery name="qry">
select * from #k# OFFSET #rowsOffset# ROWS FETCH NEXT #maxRowsPerIteration# ROWS ONLY
</cfquery>
<cfif not qry.recordCount>
<cfbreak>
</cfif>
<cfset rowsOffset += maxRowsPerIteration>
<cfloop query="qry">
<cfset rowJSON = serializeJSON(
queryRowToStruct(qry, qry.currentRow)
)>
<cfset fileAppend(k, rowJSON, "UTF-8", true)>
</cfloop>
</cfloop>
<!--- NOTE: if you locked the table previously, unlock it here --->
</cfloop>
For an reference implementation of queryRowToStruct, check CFLib.
This is really a comment, but it is way too long.
SQL Server 2017 can create JSON directly.
<cfloop list="t" index="k">
<cfquery name="qry">
SELECT (
SELECT *
FROM #k#
FOR JSON AUTO
) AS data
</cfquery>
<cffile action="write" file="#k#" output="#qry.data#">
</cfloop>
Others have touched upon the JVM and Garbage Collection but have not followed up on the potential quick win due to how CF handles GC.
CF can GC after each function returns and also at the end of each request. So if you do something that uses a lot of memory a few times in a loop, or do something that uses a moderate amount of memory a lot of times in a loop, then you should abstract that 'something' into a function, and call that function inside the loop, so that the memory can be released per iteration if necessary, instead of being held until the end of the request and potentially maxing out the heap space before the end-of-request Garbage Collection.
Eg refactoring your original code to this, is much more GC friendly:
<cffunction name="tableToFile" output="false">
<cfargument name="tableName" type="variableName" required="true" />
<cfquery name="local.qry">
select * from #arguments.tableName#
</cfquery>
<cfset local.js = serializeJSON(local.qry,'struct') />
<cffile action="write" file="#arguments.tableName#" output="#local.js#" />
</cffunction>
<cfloop list="t" index="k">
<cfset tableToFile(tableName=k) />
</cfloop>
This approach won't solve your problem though if any single iteration of that loop consumes too much memory because the query is too large. If that is your problem then you should implement this in combination with an approach like Alex's to fetch your rows in batches, and presuming your SQL Server is better up to the task than your Lucee Server, then also James' approach to let SQL Server do the JSON serialization.
Programming in Lucee (Ubuntu, Firefox) and have a query in which I need duplicate columns. In generic SQL run from my terminal this produces exactly what I ask for. It works fine in ColdFusion. But when I run it in Lucee, it refuses to accept the request for the duplicate column.
<cfoutput>
<cfset mylist = "PersonFn, PersonLn, PersonState, PersonLn">
<cfquery name = "betty" datasource = "Moxart">
select #mylist#
from Person
limit 5
</cfquery>
</cfoutput>
When I get the columnlist for this Query, it has only 3 items in it, having eliminated the duplicate column. When I run a report against the output:
<cfset m = 0>
<cfoutput query = "betty">
<cfloop list = #mylist# index = "xxcol">
<cfset m = m + 1>
#betty[xxcol][currentrow]#
<cfif m EQ 4>
<br>
</cfif>
</cfloop>
</cfoutput>
I get this error:
key [ PersonLn] not found in query, columns are [PersonFn,PersonLn,PersonState]
I really, really need that duplicate column. Programming around this would be very difficult and excruciating.
I cannot use an alias, because the actual list is chosen by the user and there is no way to know what he might choose, or how many duplicates I might have. For similar reasons, I cannot reconstruct the data after the query to create the extra columns -- since I don't know what they are.
To be more exact, I probably can do those things, but the amount and complexity of programming is significant -- and all because Lucee just won't do what generic MySql will do.
Can anyone think of a way I can get those duplicate columns back into my output without major programming?
You can't have duplicate columns in sql you have to rename it to field1 and field2 for example.
What you seek violates the Sql standard.
I have had a good search through to see if there was something similar to what I am trying to do. Nothing specifically covers it though so without further ado.
I would like my cfloop to work through any of the checked boxes on a search page and display the appropriate results drawn from the database. This is what I have so far:
<cfquery name="joblibrary">
SELECT *
FROM tblJobLibraryRoles JOIN tblJobLibraryCategories
ON tblJobLibraryRoles.category = tblJobLibraryCategories.id
<cfloop list="#form.cbGrade#" index="i">
WHERE grade=<cfqueryparam cfsqltype="cf_sql_varchar" value="#i#"/>
</cfloop>
ORDER BY category, grade, title, heraRef;
</cfquery>
Now it works all fine if only one checkbox is ticked, so I am basically asking can I get this type of CFLOOP to work in the way I've created it, or am I barking up the wrong tree?
You don't need a loop. Just use the sql keyword "in".
where somefield in (
<cfqueryparam
cfsqltype="cf_sql_varchar" value="#form.checkboxfield#"
list="yes">
)
You just have to do something to contend with the situation where no boxes are checked.
IN () is certainly the way to handle this (works with other sql statements as well), but there may be times where you want multiple WHERE conditions that are all entirely defined by variables.
Again, IN () is the solution here, but I'll demonstrate with your query on how to do what you were originally wanting to do.
Simply add a WHERE 0=0 (or 1=1 or anystring=anysamestring. You could even say Where 'trickwhere'='trickwhere' but there's no reason to get wordy with it) and loop over the rest using an AND instead.
<cfquery name="joblibrary">
SELECT * from tblJobLibraryRoles JOIN tblJobLibraryCategories ON tblJobLibraryRoles.category=tblJobLibraryCategories.id
WHERE 0=0
<cfloop list="#form.cbGrade#" index="i">AND grade=<cfqueryparam cfsqltype="cf_sql_varchar" value="#i#"/></cfloop>
ORDER BY category, grade, title, heraRef;
</cfquery>
I'm working on an existing multi-language site (Coldfusion/MySQL).
Why is it that on a lot of pages I'm sitting on, some text strings are always hard-coded into the markup like:
<CFIF language = "EN"><p>Hello World</p></CFIF>
while others use the database to update text like so:
<p><cfoutput>#tx_greetings#</cfoutput></p>
What is the best practice here? I thought if I'm going to use a database for translations, it would be easier to store all texts in there (long and small). If I'm not using a database, then all texts should be if-elsed. Mixing it is a little maintenance-heavy, isn't it?
Also, is there a limit on text-string-length, which I'm storing to MySQL? Maybe performance-wise?
Thanks for some inputs!
You shouldn't store strings/translations in your code, that's bad practice if you want a maintainable i18n'd site.
You should store all your string in the same location, db or a properties file per language. It doesn't matter which, but be consistent. Personally I prefer a properties file as its easy to edit.
welcome_message=Hi {0}, Welcome to my site
Load all your translations in one go in onApplicationStart(), then provide a wrapper class to access them and to format the string with supplied arguments
for example
#i18n.getString(user.getLocale(), "welcome_message", [user.getUsername()])#
You can use java.text.MessageFormat[1] to provide powerful formatting
function getString(string locale, string key, array args) {
var mf = createobject("java", "java.text.MessageFormat");
mf.init(variables.strings[arguments.locale][arguments.key]);
return mf.format(javacast("java.lang.Object[]", args));
}
The above is just an example, and you need to provide error catching and caching to this
Hope that helps point you in a productive direction
[1] http://docs.oracle.com/javase/7/docs/api/index.html?java/text/MessageFormat.html
You can use a DB or a ascii file, depends on which you prefer.
If u use a DB you can create a table with the following columns:
country_code : country code for the language (i.e. US for english)
definition_name : name of the definition or message (*i.e. db_error_msg for a generic error message for db action*)
definition value : value of the definition (i.e. Sorry, an error occurred saving your data)
Each record will be a definition.
Depending on the language the user select your app will filter the database and you will get a query of all definitions you need.
I usually use that query to set a session variable structure like:
<cfif IsDefined("session.language") IS FALSE>
<cfquery name="getDefinition" datasource="dsn">
SELECT * FROM tbl_definitions WHERE country_code = "US"
</cfquery>
<cfset session.language = structnew()>
<cfoutput query="getDefinitions">
<cfset session.language["#definition_name#"] = "#definition_value#">
</cfoutput>
</cfif>
In the code I will simply use:
<cfoutput>
<h2>#session.language.db_error_msg#</h2>
</cfoutput>
and I will get the right message for the current language.
You can also use a master definition db to be used by different websites.
Same solution can be used with different configuration files (ie US.cfg. EN.cfg, ES.cfg) where you set your definitions in simple way to get a list.
I usually use the following system:
definition_name = definition_value for each line
db_error_msg = Sorry, an error occured saving your data
db_success_msg = Record saved
Then I read the current language configuration file (i.e. US.cfg for english, ES.cfg for spanish) and get the same result
<cfif IsDefined("session.language") IS FALSE>
<cffile action="read" file="#path#\US.cfg" variable="definitions">
<cfset session.language = structnew()>
<cfloop index="i" list="#definitions#" delimiters="#chr(10)#">
<cfset definition_name = ListGetAt(i,1,"=")>
<cfset definition_value = ListGetAt(i,2,"=")>
<cfoutput>
<cfset session.language["#definition_name#"] = "#definition_value#">
</cfoutput>
<cfloop>
</cfif>
This can be done just when session starts (if you know the language you need) and in both ways your definitions will be available everywhere inside your application for the user session time duration you have defined.
You can use definitions for buttons, messages, table headers, etc. creating a multilanguage ui in a very fast way without creating localized templates or using inline translations.
I hope this will help you.
Just as some generic localization advice, be careful about the variable name you'll use to know which phrase to retrieve. Don't just make it the english phrase, but make it something that is clearly a specific variable because you'll also have to handle contextual phrases that seem the same in English, but are very different in other languages depending on context.