How to execute large sql query in magento? - mysql

I want to store huge content in db and my sample text is 16129 characters in length when i tried to execute this query it is showing "error:The requested URL could not be retrieved" in firefox and "no-data received" in chrome.
Moreover I use LONGTEXT as datatype for text in DB.
I also tried to execute the query directly in phpmyadmin it is working correctly.
The code is shown below.
public function _getConnection($type = 'core_write') {
return Mage::getSingleton('core/resource')->getConnection($type);
}
public function testdbAction(){
$db = $this->_getConnection();
$current_time=now();
$text="The European languages are members of the same family...... ...Europe uses the same vocabulary. The ";//text is 16129 characters in length
$sql = "INSERT into test(`usercontent_id`,`app_id`,`module_id`,`customer_id`,`content`,`created_time`,`updated_time`,`item_id`,`index_id`,`position_id`) VALUES (NULL, 15, 9,2,'" .$text. "','" . $current_time . "','" . $current_time . "',1003,5,4)";
$db->query($sql);
}
How do i handle this? any suggestions or help..

Try using $db->exec($sql) instead of $db->query($sql)

For manipulating with database structure and data Magento has a dedicated place. It is called install / upgrade scripts. It was made to keep track of modules version for easy updates. So you should use a script to add new data.
Here's an example.
config.xml of your module:
<modules>
<My_Module>
<version>0.0.1</version>
</My_Module>
</modules>
<global>
<resources>
<my_module_setup>
<setup>
<module>My_Module</module>
<class>Mage_Core_Model_Resource_Setup</class>
</setup>
</my_module_setup>
</resources>
</global>
Now, you need to create following file:
My_Module/sql/my_module_setup/install-0.0.1.php
Now, depending on your Magento version, file would need to have a different name. If you're using Magento CE 1.4 or lower (EE 1.8) then you should name it mysql4-install-0.0.1.php.
This script will be launched at next website request. Class Mage_Core_Model_Resource_Setup will execute code inside install-0.0.1.php. From within the install script you will have access to the Setup class by using $this object;
So now, you can write in script following code:
$this->startSetup();
$text = <<<TEXT
YOUR LONG TEXT GOES HERE
TEXT;
$this->getConnection()
->insert(
$this->getTable('test'),
array(
'usercontent_id' => null,
'app_id' => 15,
'content' => $text,
//other fields in the same fashion
)
);
$this->endSetup();
And that's it. It's a clean and appropriate way of adding custom data to the database in Magento.
If you want to save a user input on a regular basis using forms, then I recommend creating w model, resource model and collection, define entities in config.xml. For more information please refer to Alan Storm's articles, like this one.
I hope that I understood your question correctly.

Related

Using wpdb to display non-wordpress database data not working

I have created a section in my functions.php file and registered a shortcode which is intended to echo some data from a custom mysql database table.
I have an external system which will insert data into the database.
My wordpress install should be able to read from this database, and I would like to echo some rows on a wordpress page via shortcode.
The problem I am having is that the table data is not being echoed out. I can do echo "test"; and the word test will get displayed but the problem I having is not showing table data. Heres my code.
function vstp_feed_function(){
//$x++;
//$alternatingclass = ($x%2 == 0)? 'whiteBackground': 'graybackground';
$mydb = new wpdb('root','cencored','testdb','localhost');
$rows = $mydb->get_results("select * from services");
echo "<ul>";
foreach ($rows as $obj) :
echo "<li>".$obj->headcode."</li>";
endforeach;
echo "</ul>";
}
add_shortcode('vstp_feed', 'vstp_feed_function');
So the above code will create a new variable mydb, with the database credentials and information. A new variable called $rows will store the sql query which uses the mydb variable. I get all columns from services table. Thereafter I am echoing an unordered list, running a for loop to display the contents of the column "headcode" in list item html.
Once the select query is done, the for loop ends and so does the html list.
The shortcode is registered as [vstp_feed] which is tested and working with static html.
Any help would be greatly appreciated. Worked with sql before but not with new wordpress db instances. Thanks.
Just to confirm, its apache ubuntu. Its also worth noting that my ##datadir is stored on a different disk seperate from my OS disk, for faster IO and freedom to expand storage on the fly
Place your connection code inside wp-config.php file ( $mydb = new wpdb('root','cencored','testdb','localhost'); )
And use it in your function file as "global $mydb;"

find the path to the script which is running the query

is there any query to return information about the current script that is running it ?
like for example if i have a file in
/home/domain/public_html/script.php
i want to put a query in it that file to return the filename and path i.e :
/home/domain/public_html/script.php
or at least the base path to it ]
/home/domain/public_html/
pleas note i know lots of method to do this but i specifically want to get these information back from database in response and after running a query
No, there is no way for the MySQL Server to know the name or path to the PHP script that is executing queries unless you tell it.
I've seen some projects that establish a coding practice to append a comment to the SQL queries with information to help you identify the source.
SELECT * FROM MyTable WHERE id = 123; /* File: script.php, Function: myFunction() */
You then see these comments appearing in the MySQL processlist, and in query logs.
You have to write code to put the comment into the SQL yourself:
$sql = sprintf("SELECT * FROM MyTable WHERE id = 123; /* File: %s, Function: %s() */",
__FILE__, __FUNCTION__);
If you don't want to change the code, another great solution is to use New Relic Application Monitoring, which tracks SQL queries in code for you, without any need to modify your code. It's costly to pay for the New Relic service, but it's the best solution. If you want to explore cheaper alternatives, search google "alternatives to new relic".

EntityFramework on MySql changing connection string does not change results data

I'm working with EntityFramework 5.0 and MySql. I have generated model from database, and my application now have to connect on multiple database with same structred data.
So i have to dynamic change connection string based on some info.
I try to change database name even from config section of connection string, and with EntityConnectionStringBuilder, but i had the same result: my new connection is stored correctly, but data returned are of the first database.
From WebConfig:
add name="dbIncassiEntities" connectionString="metadata=res:///DAL.Modelincassi.csdl|res:///DAL.Modelincassi.ssdl|res://*/DAL.Modelincassi.msl;provider=Devart.Data.MySql;provider connection string="user id=root ... database=dbname2"" providerName="System.Data.EntityClient" />
From code:
EntityConnectionStringBuilder entityBuilder = new EntityConnectionStringBuilder();
entityBuilder.Provider = providerName;
entityBuilder.ProviderConnectionString = "user id=...database=dbname2";
entityBuilder.Metadata = #"res://*/DAL.Modelincassi.csdl|res://*/DAL.Modelincassi.ssdl|res://*/DAL.Modelincassi.msl";
var context = new dbIncassiEntities(entityBuilder.ToString());
My constructor:
public dbIncassiEntities(string conn)
: base(conn)
{
}
What am i missing?
UPDATE
I can see that calling a query directly from SqlQuery, results returned are correct,
while using the generated entities i retrieve wrong data.
var test = context.Database.SqlQuery<string>(
"SELECT cognomenome FROM addetto limit 0,1").ToList();
But calling..
var oAddetto = from c in context.addettoes select c;
So my problem is only on the model itsself, and manually changing the generated schema
<EntitySet Name="addetto" EntityType="dbIncassiModel.Store.addetto" store:Type="Tables" Schema="dbname2" />
..i'll get the right information.
My question now is: how can i change in code these informations??
Any help is really appreciated!!
Thanks, David
Ok, i've found a workaround for now.
I simply clear the shema name on the designer, and now i can call the generated entities succesfully. Hope this can help anyone else.
David
While I could not remove the Schema in the designer, I removed it directly in the .edmx file. Do a full text search for Schema="YourSchema" in an XML editor of your choice and remove the entries. After that, changing the connection string is enough.
Downside is, the Visual Studio designer and mapping explorer won't work properly anymore.
This seems to be more of a dotConnect issue rather than MySQL, since the problem also exists for the Oracle adapter:
http://forums.devart.com/viewtopic.php?t=17427

Pairing content on an external website with entries in an mySQL database

tl;dr: I'm looking for a way to find entries in our database which are missing information, getting that information from a website and adding it to the database entry.
We have a media management program which uses a mySQL table to store the information. When employees download media (video files, images, audio files) and import it into the media manager they are suppose to also copy the description of the media (from the source website) and add it to the description in the Media Manager. However this has not been done for thousands of files.
The file name (eg. file123.mov) is unique and the details page for that file can be accessed by going to a URL on the source website:
website.com/content/file123
The information we want to scrape from that page has an element ID which is always the same.
In my mind the process would be:
Connect to database and Load table
Filter: "format" is "Still Image (JPEG)"
Filter: "description" is "NULL"
Get first result
Get "FILENAME" without extension)
Load the URL: website.com/content/FILENAME
Copy contents of the element "description" (on website)
Paste contents into the "description" (SQL entry)
Get 2nd result
Rinse and repeat until last result is reached
My question(s) are:
Is there software that could perform such a task or is this something that would need to be scripted?
If scripted, what would be the best type of script (eg could I achieve this using AppleScript or would it need to be made in java or php etc.)
Is there software that could perform such a task or is this something that would need to be scripted?
I'm not aware of anything that will do what you want out of the box (and even if there was, the configuration required won't be much less work than the scripting involved in rolling your own solution).
If scripted, what would be the best type of script (eg could I achieve this using AppleScript or would it need to be made in java or php etc.)
AppleScript can't connect to databases, so you will definitely need to throw something else into the mix. If the choice is between Java and PHP (and you're equally familiar with both), I'd definitely recommend PHP for this purpose, as there will be considerably less code involved.
Your PHP script would look something like this:
$BASEURL = 'http://website.com/content/';
// connect to the database
$dbh = new PDO($DSN, $USERNAME, $PASSWORD);
// query for files without descriptions
$qry = $dbh->query("
SELECT FILENAME FROM mytable
WHERE format = 'Still Image (JPEG)' AND description IS NULL
");
// prepare an update statement
$update = $dbh->prepare('
UPDATE mytable SET description = :d WHERE FILENAME = :f
');
$update->bindParam(':d', $DESCRIPTION);
$update->bindParam(':f', $FILENAME);
// loop over the files
while ($FILENAME = $qry->fetchColumn()) {
// construct URL
$i = strrpos($FILENAME, '.');
$url = $BASEURL . (($i === false) ? $FILENAME : substr($FILENAME, 0, $i));
// fetch the document
$doc = new DOMDocument();
$doc->loadHTMLFile($url);
// get the description
$DESCRIPTION = $doc->getElementsById('description')->nodeValue;
// update the database
$update->execute();
}
I too am not aware of any existing software packages that will do everything you're looking for. However, Python can connect to your database, make web requests easily, and handle dirty html. Assuming you already have Python installed, you'll need three packages:
MySQLdb for connecting to the database.
Requests for easily making http web requests.
BeautifulSoup for robust parsing of html.
You can install these packages with pip commands or Windows installers. Appropriate instructions are on each site. The whole process won't take more than 10 minutes.
import MySQLdb as db
import os.path
import requests
from bs4 import BeautifulSoup
# Connect to the database. Fill in these fields as necessary.
con = db.connect(host='hostname', user='username', passwd='password',
db='dbname')
# Create and execute our SELECT sql statement.
select = con.cursor()
select.execute('SELECT filename FROM table_name \
WHERE format = ? AND description = NULL',
('Still Image (JPEG)',))
while True:
# Fetch a row from the result of the SELECT statement.
row = select.fetchone()
if row is None: break
# Use Python's built-in os.path.splitext to split the extension
# and get the url_name.
filename = row[0]
url_name = os.path.splitext(filename)[0]
url = 'http://www.website.com/content/' + url_name
# Make the web request. You may want to rate-limit your requests
# so that the website doesn't get angry. You can slow down the
# rate by inserting a pause with:
#
# import time # You can put this at the top with other imports
# time.sleep(1) # This will wait 1 second.
response = requests.get(url)
if response.status_code != 200:
# Don't worry about skipped urls. Just re-run this script
# on spurious or network-related errors.
print 'Error accessing:', url, 'SKIPPING'
continue
# Parse the result. BeautifulSoup does a great job handling
# mal-formed input.
soup = BeautifulSoup(response.content)
description = soup.find('div', {'id': 'description'}).contents
# And finally, update the database with another query.
update = db.cursor()
update.execute('UPDATE table_name SET description = ? \
WHERE filename = ?',
(description, filename))
I'll warn that I've made a good effort to make that code "look right" but I haven't actually tested it. You'll need to fill in the private details.
PHP is a good scraper . I have made a class that wraps the cURL port of PHP here:
http://semlabs.co.uk/journal/object-oriented-curl-class-with-multi-threading
You'll probably need to use some of the options:
http://www.php.net/manual/en/function.curl-setopt.php
To scrape HTML, I usually use regular expressions, but here is a class I made that should be able to query HTML without issues:
http://pastebin.com/Jm9jKjAU
Useage is:
$h = new HTMLQuery();
$h->load( $string_containing_html );
$h->getElements( 'p', 'id' ); // Returns all p tags with an id attribute
The best option to scrape would be XPath, but it can't handle dirty HTML. You can use that to do things like:
//div[#class = 'itm']/p[last() and text() = 'Hello World'] <- selects the last p in div elements that have the innerHTML 'Hello World'
You can use that in PHP with the DOM class (built-in).

How to covert a mySql DB into Drupal format tables

Hi there i have some sql tables and i want to convert these in a "Drupal Node Format" but i don't know how to do it. Does someone knows at least which tables i have to write in order to have a full node with all the keys etc. ?
I will give an example :
I have theses Objects :
Anime
field animeID
field animeName
Producer
field producerID
field producerName
AnimeProducers
field animeID
field producerID
I have used the CCK module and i had created in my drupal a new Content Type Anime and a new Data Type Producer that exist in an Anime object.
How can i insert all the data from my simple mysql db into drupal ?
Sorry for the long post , i would like to give you the chance to understand my problem
Thx in advance for your time to read my post
You can use either the Feeds module to import flat CSV files, or there is a module called Migrate that seems promising (albiet pretty intense). Both work on Drupal 6 or 7.
mmmmm.... i think you can export CVS from your sql database and then use
http://drupal.org/project/node_import
to import this cvs data to nodes.....mmmm i don know if there is another non-programmatically way
The main tables for node property data are node and node_revision, have a look at the columns in those and it should be fairly obvious what needs to go in those.
As far as fields go, their storage is predictable so you would be able automate an import (although I don't envy you having to write that!). If your field is called 'field_anime' it's data will live in two tables: field_data_field_anime and field_revision_field_anime which are keyed by the entity ID (in this case node ID), entity type (in the case 'node' itself) and bundle (in this case the name of your node type). You should keep both tables up to date to ensure the revision system functions correctly.
The simplest way to do it though is with PHP and the node API functions:
/* This is for a single node, obviously you'd want to loop through your custom SQL data here */
$node = new stdClass;
$node->type = 'my_type';
$node->title = 'Title';
node_object_prepare($node);
// Fields
$node->field_anime[LANGUAGE_NONE] = array(0 => array('value' => $value_for_field));
$node->field_producer[LANGUAGE_NONE] = array(0 => array('value' => $value_for_field));
// And so on...
// Finally save the node
node_save($node);
If you use this method Drupal will handle a lot of the messy stuff for you (for example updating the taxonomy_index table automatically when adding a taxonomy term field to a node)