I've been building a new coldfusion application leveraging object oriented approach and stored procedures. While everything works like a charm, the app is very fast due to SPs and optimized mysql code, I would really appreciate if you would help me clear one thing up :) I have dbcode.cfc which as you probably already guessed stores all queries with stored procedures. That said, generally what's the best approach when it comes to storing global DNS parameters?
1.this can be used for one global DSN
<cfset this.datasource ="myDB">
2.this can also be used for one global DSN
<cffunction name="onApplicationStart" returnType="boolean" output="false">
<cfset application.dsn = "myDB">
<cfset application.username = "userName">
<cfset application.password = "password">
<cfreturn true>
</cffunction>
2.1 everything as above but onRequest
3.in my case I could also create a global variables within the dbcode.cfc
<cfset variables.dsn = "myDB">
<cfset variables.username = "userName">
<cfset variables.password = "password">
4.additionally one could use something like this for setting multiple datasources
<cffunction name="onApplicationStart" returnType="boolean" output="false">
<cfset application.myDSNs = StructNew()>
<cfset application.myDSNs.1 = "myDB1">
<cfset application.myDSNs.2 = "myDB2">
.
.
.
<!---something similar for usernames and passwords--->
</cffunction>
so what's the most efficient way to handle this sort of thing?
Let me see if I can answer each part of your DSN questions.
Setting the datasource as a part the application.cfc constructor is OK
Setting up the DSN in OnApplicationStart() works too, but you need to use:
<cfset application.datasource = "myDB">
2.1 Setting the datasource on every request is pointless, unless you datasource changes with every request. I suspect is does not
You should not touch queries.cfc. That is internal to ColdFusion and has the potential to break everything on your server.
You may want to read on up on the difference between arrays and structs. It looks like what you wanted to do was:
<cfset application.myDSNs = ["MyDB1", "MyDB2]>
They would then be accessable via:
#application.myDSNs[1]#
#application.myDSNs[2]#
I would not do this approach. Databases are typically not arrays of anything. Each server has its own purpose. You DSNs should be reflective of one of more of the following:
The Server
The Database
Purpose
Related
I had this code which works fine when the database is small with few records, it writes the json to the file properly, but when the data is huge, it just times out
<cfloop list="t" index="k">
<cfquery name="qry">
select * from #k#
</cfquery>
<cfset js= serializeJSON(qry,'struct')>
<cffile action="write" file="#k#" output="#js#">
</cfloop>
I tried using threads but they are also not working, it just creates empty tables files with no values if i use cfthread with joins
Thought of splitting the files into a combination of 1000 records for each table and and then doing like
table_1, table2, table3, of the same table which is table because it has millions of records and skip for those if they have less than 1000 records to create only 1 file.
but i am just thinking which approach is best and a starting pointing is needed
First of all, let's split this up:
Resultset from database
<cfquery name="qry">
select * from #k#
</cfquery>
Database server retrieves data and streams it via network to the ColdFusion server
ColdFusion stores the data in a query object and stores it in the heap
Serializing the resultset from database
<cfset js= serializeJSON(qry,'struct')>
ColdFusion recursively serializes the whole query object
ColdFusion creates a string object that contains the serialized data and stores it in the heap
Writing the serialized resultset from memory onto the filesystem
<cffile action="write" file="#k#" output="#js#">
ColdFusion writes the string object into a file on the filesystem
Doing all of this within the same request/thread
<cfloop list="t" index="k">
...
</cfloop>
Conclusion
Your code tortures the JVM heap, because references have to be kept until the end of each iteration. The GC can only clean up after a full table has been processed. Large tables (1.000.000+ rows) will likely kill the thread or even hang the JVM.
The Fix: Resultset from database
Retrieving large resultsets at once will always hurt performance. While streaming lots of data within a local network (assuming the database is in the same network) just takes a bit more time, the memory required to store the full resultset is going to be an issue for the JVM.
Instead of doing everything at once, consider splitting it up in smaller chunks of data. Use OFFSET and FETCH in the SQL statement to limit the number of rows per loop. Having multiple iterations will allow the Java GC to free up memory used by previous iterations, relieving the heap.
The Fix: Serializing the resultset from database
Same issue. Big datasets whill hurt performance. Split the resultset by serializing row by row instead of all rows at once.
Writing the serialized resultset from memory onto the filesystem
While this one probably doesn't need a fix, you eventually have to switch to writing line after line.
Some code
<cfset maxRowsPerIteration = 50000>
<cfloop list="t" index="k">
<!--- create empty file to append lines later --->
<cfset fileWrite(k, "")>
<cfset rowsOffset = 0>
<!--- NOTE: you might want to lock the table (prevent write access) here --->
<!--- infinite loop will be terminated as soon the query no longer returns any rows --->
<cfloop condition="true">
<!--- fetch a slice of the full table --->
<cfquery name="qry">
select * from #k# OFFSET #rowsOffset# ROWS FETCH NEXT #maxRowsPerIteration# ROWS ONLY
</cfquery>
<cfif not qry.recordCount>
<cfbreak>
</cfif>
<cfset rowsOffset += maxRowsPerIteration>
<cfloop query="qry">
<cfset rowJSON = serializeJSON(
queryRowToStruct(qry, qry.currentRow)
)>
<cfset fileAppend(k, rowJSON, "UTF-8", true)>
</cfloop>
</cfloop>
<!--- NOTE: if you locked the table previously, unlock it here --->
</cfloop>
For an reference implementation of queryRowToStruct, check CFLib.
This is really a comment, but it is way too long.
SQL Server 2017 can create JSON directly.
<cfloop list="t" index="k">
<cfquery name="qry">
SELECT (
SELECT *
FROM #k#
FOR JSON AUTO
) AS data
</cfquery>
<cffile action="write" file="#k#" output="#qry.data#">
</cfloop>
Others have touched upon the JVM and Garbage Collection but have not followed up on the potential quick win due to how CF handles GC.
CF can GC after each function returns and also at the end of each request. So if you do something that uses a lot of memory a few times in a loop, or do something that uses a moderate amount of memory a lot of times in a loop, then you should abstract that 'something' into a function, and call that function inside the loop, so that the memory can be released per iteration if necessary, instead of being held until the end of the request and potentially maxing out the heap space before the end-of-request Garbage Collection.
Eg refactoring your original code to this, is much more GC friendly:
<cffunction name="tableToFile" output="false">
<cfargument name="tableName" type="variableName" required="true" />
<cfquery name="local.qry">
select * from #arguments.tableName#
</cfquery>
<cfset local.js = serializeJSON(local.qry,'struct') />
<cffile action="write" file="#arguments.tableName#" output="#local.js#" />
</cffunction>
<cfloop list="t" index="k">
<cfset tableToFile(tableName=k) />
</cfloop>
This approach won't solve your problem though if any single iteration of that loop consumes too much memory because the query is too large. If that is your problem then you should implement this in combination with an approach like Alex's to fetch your rows in batches, and presuming your SQL Server is better up to the task than your Lucee Server, then also James' approach to let SQL Server do the JSON serialization.
Question: What data type do I use when invoking JSON data from a database using ColdFusion?
Background: My application needs to pull some JSON data from a database and parse the data into an HTML form.
Below is my code so far:
<cftry>
<cfinvoke component="UserData.cfc.data" method="editData" returnvariable="editReturn">
<cfinvokeargument name="formID" value="#URL.dataID#">
</cfinvoke>
<cfset ReBuild = DeserializeJSON(#editReturns#)>
<cfcatch type="Any">
<cfoutput>
<hr>
<h1>Other Error: #cfcatch.Type#</h1>
<ul>
<li><b>Message:</b> #cfcatch.Message#
<li><b>Detail:</b> #cfcatch.Detail#
</ul>
</cfoutput>
<cfset errorCaught = "General Exception">
</cfcatch>
</cftry>
UserData.cfc.data:
<cffunction name="editData" access="public" returntype="any">
<cfargument name="formID" required="yes">
<!--- Select --->
<cfquery name="UserData" datasource="RC">
SELECT Data
FROM USER_Forms
WHERE ID = <cfqueryparam value="#ARGUMENTS.formID#">
</cfquery>
<!-- The information pulled from the database should be a Serialized JSON data. -->
<cfreturn UserData>
</cffunction>
Error Message:
Other Error: Expression
Message: Complex object types cannot be converted to simple values.
Detail: The expression has requested a variable or an intermediate expression result as a simple value. However, the result cannot be converted to a simple value. Simple values are strings, numbers, boolean values, and date/time values. Queries, arrays, and COM objects are examples of complex values.
The most likely cause of the error is that you tried to use a complex value as a simple one. For example, you tried to use a query variable in a cfif tag.
When I added the data to the database I used the following process:
<cfset ForDBInsert = SerializeJSON(formCopy)>
<!-- Then I INSERTED ForDBInsert into the database columnn. -->
Try DeserializeJSON(editReturns.data) - notice I took out the # they are not needed when passing arguments this way. Looks like you are trying to deserialize the entire query object rather than the string itself.
I'm working on an existing multi-language site (Coldfusion/MySQL).
Why is it that on a lot of pages I'm sitting on, some text strings are always hard-coded into the markup like:
<CFIF language = "EN"><p>Hello World</p></CFIF>
while others use the database to update text like so:
<p><cfoutput>#tx_greetings#</cfoutput></p>
What is the best practice here? I thought if I'm going to use a database for translations, it would be easier to store all texts in there (long and small). If I'm not using a database, then all texts should be if-elsed. Mixing it is a little maintenance-heavy, isn't it?
Also, is there a limit on text-string-length, which I'm storing to MySQL? Maybe performance-wise?
Thanks for some inputs!
You shouldn't store strings/translations in your code, that's bad practice if you want a maintainable i18n'd site.
You should store all your string in the same location, db or a properties file per language. It doesn't matter which, but be consistent. Personally I prefer a properties file as its easy to edit.
welcome_message=Hi {0}, Welcome to my site
Load all your translations in one go in onApplicationStart(), then provide a wrapper class to access them and to format the string with supplied arguments
for example
#i18n.getString(user.getLocale(), "welcome_message", [user.getUsername()])#
You can use java.text.MessageFormat[1] to provide powerful formatting
function getString(string locale, string key, array args) {
var mf = createobject("java", "java.text.MessageFormat");
mf.init(variables.strings[arguments.locale][arguments.key]);
return mf.format(javacast("java.lang.Object[]", args));
}
The above is just an example, and you need to provide error catching and caching to this
Hope that helps point you in a productive direction
[1] http://docs.oracle.com/javase/7/docs/api/index.html?java/text/MessageFormat.html
You can use a DB or a ascii file, depends on which you prefer.
If u use a DB you can create a table with the following columns:
country_code : country code for the language (i.e. US for english)
definition_name : name of the definition or message (*i.e. db_error_msg for a generic error message for db action*)
definition value : value of the definition (i.e. Sorry, an error occurred saving your data)
Each record will be a definition.
Depending on the language the user select your app will filter the database and you will get a query of all definitions you need.
I usually use that query to set a session variable structure like:
<cfif IsDefined("session.language") IS FALSE>
<cfquery name="getDefinition" datasource="dsn">
SELECT * FROM tbl_definitions WHERE country_code = "US"
</cfquery>
<cfset session.language = structnew()>
<cfoutput query="getDefinitions">
<cfset session.language["#definition_name#"] = "#definition_value#">
</cfoutput>
</cfif>
In the code I will simply use:
<cfoutput>
<h2>#session.language.db_error_msg#</h2>
</cfoutput>
and I will get the right message for the current language.
You can also use a master definition db to be used by different websites.
Same solution can be used with different configuration files (ie US.cfg. EN.cfg, ES.cfg) where you set your definitions in simple way to get a list.
I usually use the following system:
definition_name = definition_value for each line
db_error_msg = Sorry, an error occured saving your data
db_success_msg = Record saved
Then I read the current language configuration file (i.e. US.cfg for english, ES.cfg for spanish) and get the same result
<cfif IsDefined("session.language") IS FALSE>
<cffile action="read" file="#path#\US.cfg" variable="definitions">
<cfset session.language = structnew()>
<cfloop index="i" list="#definitions#" delimiters="#chr(10)#">
<cfset definition_name = ListGetAt(i,1,"=")>
<cfset definition_value = ListGetAt(i,2,"=")>
<cfoutput>
<cfset session.language["#definition_name#"] = "#definition_value#">
</cfoutput>
<cfloop>
</cfif>
This can be done just when session starts (if you know the language you need) and in both ways your definitions will be available everywhere inside your application for the user session time duration you have defined.
You can use definitions for buttons, messages, table headers, etc. creating a multilanguage ui in a very fast way without creating localized templates or using inline translations.
I hope this will help you.
Just as some generic localization advice, be careful about the variable name you'll use to know which phrase to retrieve. Don't just make it the english phrase, but make it something that is clearly a specific variable because you'll also have to handle contextual phrases that seem the same in English, but are very different in other languages depending on context.
Stupid problem, I have a function that runs a query several times in a block & reports/emails if/when a query fails. I'm wondering if/how I can capture the actual MySQL error & return it as part of my email/report.
So far I see no way to do this.
Any thoughts?
-thanks
-sean
UPDATE
Thanks Charlie;
I never considered using the cfcatch structure [and truthfully didn't realize it returned so much useful stuff!!]
Unfortunately the host does not allow cfdump so I had to go about it like this:
<cftry>
<some sql>
<cfcatch type="any">
<cfscript>
for (key in cfcatch) {
try{
variables.report = variables.report&"<li>"&key&"="&cfcatch[key]&"</li>";
}
catch(Any excpt) {
variables.report = variables.report&"<li>"&key&"=??</li>";
}
}
</cfscript>
<cfcatch>
<cftry>
Isn't the native database error returned as part of the cfcatch?
<cftry>
(some sql here)
<cfcatch type="any">
<cfdump var="#cfcatch#" />
</cfcatch>
</cftry>
If you run that on a page, and intentionally use some invalid SQL, what do you see in the cfdump?
I have a problem with saving a huge amount of records to database using CFWheels. Here is an example:
<cfloop from="1" to="10000" index="i">
<cfset var newUser = model("user").new()>
<cfset newUser.name = "Test"&i>
<cfset newUser.save()>
</cfloop>
This causes java.lang.OutOfMemoryError
Please help me how to solve this problem.
Looping over multiple database calls leading to OOM is a known ColdFusion bug. Fortunately, there is a workaround, use <cfthread/>. You should be able to alter your code as such:
<cfloop from="1" to="10000" index="i">
<cfset threadName = "thread" & createUuid()>
<cfthread name="#threadName#">
<cfset var newUser = model("user").new()>
<cfset newUser.name = "Test"&i>
<cfset newUser.save()>
</cfthread>
<cfthread action="join" name="#threadName#">
</cfloop>
In this situation, you're using the thread solely for its side effect, running in a different context so that it doesn't get retained on the heap. Thus the immediate join right after declaring the thread, so it's not actually running anything in parallel.
You can try to run the Garbage collector:
http://www.beetrootstreet.com/blog/index.cfm/2009/6/25/Clearing-ColdFusion-memory-using-garbage-collection-when-memory-gets-low
There are a couple fairly inefficient things going on here. First, it's generating 1,000 user objects, which isn't really a good idea to do in a single request in ColdFusion. Second, it's running 1,000 database queries, which isn't really a good idea to do in any programming language.
I would stop using model objects for a case like this and figure out how to condense the logic into a single database query. The ORM stuff in Wheels is generally very useful, but it has its limits in situations like this one.
For example, if you're using SQL Server 2008, you can do this inside your user model to keep everything under a single call to cfquery:
<cffunction name="batchCreate">
<cfquery datasource="#get('dataSourceName')#">
INSERT INTO
#this.tableName()# (#this.columnNameForProperty("name")#)
VALUES
<cfloop from="1" to="10000" index="i">
(<cfqueryparam cfsqltype="cf_sql_varchar" value="Test#i#">)
<cfif i lt 10000>,</cfif>
</cfloop>
</cfquery>
</cffunction>
Of course, the query will look different if you're using MySQL or another database engine.