I need to maintain an application of mass email sender. The last programmer did a nice job, but the boss feel that a little optimizations could be done on the database treatment. When a campaing is finished, the report gives the option to save the selected segment. For instance, we send 50000 emails, and we want to save the segment of people who open the newsletter (2000) The tool now creates a new segment duplicating the contacts (with an INSERT), but I think we could improve the tool by saving the id of each contact.
I would like to know if saving the contacts whith an sql IN statement would increase the performance of the tool, or there is another way to perform this. Something like:
Create a list of the ids of the contacts SELECT * FROM contacts
SELECT * FROM contacts WHERE idContact IN (all_contacts_comma_separated) --> I would save
this
Thanks in advance
PD: It's a production environment, so I need to be sure before made any changes :-(
You didn't say where the list of people who opened the email currently resides. If it's not in the database what code/process will you use to generate your IN statement list? If it is in the database why not JOIN your tables to get the information?
Either way I'd not recommend using IN when you have 2000 items in the list.
It might also be worth you reading the following:
SQL Server: JOIN vs IN vs EXISTS - the logical difference
It's written with SQL server in mind (and I'm not sure it all directly applies to MySQL) but the concepts are interesting and you should perform testing before changing your production environment, as eggyal's comment suggested.
Related
We are updating table XYZ have following fields:
First Name|Middle Name|Last Name|Address|DOB|Country|County|(etc.)
Initially, we are calling some web service which is sending updated information for a row in XYZ like either update first name or DOB update or both or all or none.
Now there is requirement to create a log table in database which store summary of old records and changes done to XYZ. Every affected row should be reported.
Is it good to create similar fields in new table say ABC:
First Name|Middle Name|Last Name|Address|DOB|Country|County|Update_Date
with additional field called "Update_datetime"
Now each time service called we will select values from previous row i.e from XYZ and update the same to ABC with update date.
What are loopholes in this practice? What other better practices can be followed?
Is there a requirement for a log table or a requirement for a proper history?
Oracle has history functionality Out of the box
I doubt MySQL does - you may have to do a different way.
The pros of Oracle is that it will not fail - it's a core feature. The cons of hand rolled is, well, it's hand rolled. Lots of SPs, triggers or other nastiness that people can deliberately or inadvertently bypass.
I echo the need to know what the requirements are behind this. Is it to be human readable (auditing, debugging etc.) or machine readable (e.g. event sourcing architectural pattern)? How often will you need to go back to look at previous versions? How often do things change?
If it's event sourcing, then there are a few answers around on Stack Overflow about that, e.g. Using an RDBMS as event sourcing storage and best event sourcing db strategy. For more of an introduction, there's e.g. a Martin Fowler video.
There are also SO answers on logging changes in MySQL and Using MySQL triggers to log all table changes to a secondary table and an alternative approach (using 1 table, but adding sort-of version numbers to show each record's validity).
Unfortunately, I have to deal with a lot of user submitted data, text fields rather than option boxes. I have imported it into my MySQL database as strings. I do all this to be able to run statistics quickly on the data like top 10 most common companies. The problem I have run into is that some of the rows have slightly different names for the same companies. For example:
Brasfield & Gorrie, LLC VS Brasfield and Gorrie
Britt Peters and Associates VS Britt, Peters & Associates Inc.
Is there some fairly straightforward MySQL command or external tool that will allow me to go through and combine these sort of rows. I know how to use REPLACE(), but I don't think it has the power to do this simply. Correct me if I'm wrong!
Taking this example:
Brasfield & Gorrie, LLC VS Brasfield and Gorrie
Assuming that I want to keep the first one, I would find all records that have the ID of the second one and update them to use the first, assuming that this table that has these titles also has an ID field for each one.
You would create a page in PHP that will allow you to administer this with mouse clicks, but it will require regular pruning since you allow users to enter this data. For future entries, you can try to apply the Levenshtein Distance and try to provide a suggestion based on available similar matches so that you can help guide the users to something that already exists rather than a new db entry.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist
Closed 9 years ago.
Improve this question
My app involves implementing data and displaying it to the user.i'll give you a full explanation of the app and what i'm trying to achieve.
Basically the app finds out what car is best suited for the user. last week i decided that a drop down menu will be easier and more interactive than a normal questionnaire. i have three labels, performance, luxury, eco friendly and three buttons, high performance, high luxury and high eco friendly. how can i add data to these so that when the user picks these preferences, it links to a specific car, let me remind you that this is just one of many controllers and questions.
I read that MYSQL is good but I'm not sure if its easier or harder. To be honest, im still in beginner level for programming. Also some linking and database videos on youtube are a joke.
I'll try to get you more on track. Your question shows that you're a newbie but if you're a newbie with some sense you'll catch on instead of complain.
MySQL is a variant of SQL. Unfortunately Oracle (evil Microsoft of the business world--think vendor lock-in) bought out Sun Microsystems (a not evil corporation that should have been defended to the death or close-to). There is some political concern that MySQL is slowly being closed off in favor of Oracle so if you have the option opt for PostgreSQL.
With a database you have two things: a database and a user.
The database part is easy, that is where information is stored.
The user part is still pretty easy, different users have different permissions. Those permissions in example are what SQL commands can be executed.
A database's structure is basically....
Database --> Table --> Columns/Rows
A database table is like an HTML table, columns are sets of data (a 'name' column, an 'address' column, etc). Each row is an entry. Bob's information may be on row 34, Sally's information on row 35, etc.
The four most common commands you'll use: SELECT, INSERT, DELETE and UPDATE. With my software unless a webmaster is signed in a lower level privileged SQL user only has access to these four base SQL commands.
There are two contexts to working with SQL: building it and using it. I'm not an expert though what I've generally accepted is that using SQL across different variants (e.g. MySQL and PostgreSQL) for day-to-day use (accessing, updating, removing, editing data) utilizes standardized SQL commands. Sure, some companies may change the structure of their database on a daily or even hourly basis ON a regular basis but generally speaking not everyone. Commands to change a database structure such as ALTER may NOT be cross-variant standard. Why does this matter? If you don't understand the politics of what is involved then YOU DO NOT UNDERSTAND WHAT IS INVOLVED. So I've made sure I'll be able to easily migrate from MySQL to PostgreSQL by avoiding vendor specific syntax. Spend an extra 20% doing things right the first time and you'll save 80% of the time wasted on maintenance by everyone else.
If you want jump in and start using MySQL get a copy of XAMPP. It takes minimal effort and is great for LOCAL testing (local means YOUR computer only accessible to you/your LAN, live means available on the web in general). I'm not sure about xcode but the general instructions should at least give you a basic guide to stay on track in regards to databases.
If you're looking to learn with a GUI you'll want to use phpMyAdmin (included with XAMPP) though I don't recommend downloading the latest copy as they're going way overboard with some things in phpMyAdmin. PostgreSQL has it's equivalent. It'll show you working (albeit VERY messy) examples of commands. For your reference here are some commands I use in the console...
The highest privileged user in SQL is usually called 'root' (without quotes). You'll have root access on your localhost (local) and dedicated servers (likely VPS too) though not on shared servers unless you're some kind of 1337 haxorz.
The four most common commands...
Using SELECT to retrieve data...
SELECT * FROM database_table;
SELECT id, name FROM database_table WHERE cool_factor='exact matching text';
Using DELETE to remove a record...
DELETE FROM database_table WHERE id='1';
Using INSERT to add a single new record...
INSERT INTO email_filters (age, name) VALUES ('Bob');
Using INSERT to add multiple new records...
INSERT INTO email_filters (name) VALUES ('Bob','23'), ('Sally','24'), ('Susan','25'), ('Irate Yeti','9001');
Using UPDATE to update an existing record...
UPDATE accounts SET name='Mr. Yeti', profession='eating people' WHERE id='234';
BTW it's common practice to make your syntax uppercase to differentiate between SOL-specific syntax and your content.
Important commands that help you create and get around in SQL...m
Login: this will prompt you for a password...
mysql -uroot -p
Show a list of databases (phpMyAdmin will let you create databases)...
show databases;
Select a database to work with...
USE mysql;
Select an SQL user...
SELECT User FROM user;
SELECT User FROM mysql.user;
Create a user (with a normal and admin example)...
CREATE USER 'example_v10'#'localhost' IDENTIFIED BY 'my_!1337_p#ss';
CREATE USER 'example_v10a'#'localhost' IDENTIFIED BY 'my_!1337_p#ss';
Before an SQL user can access a database it must be granted permission, here is a limited permission example...
GRANT SELECT, INSERT, UPDATE, DELETE ON `example_mysite_database`.* TO 'example_v10'#'localhost';
...and an administrative user being granted all permissions (use with caution)...
GRANT ALL PRIVILEGES ON `example_mysite_database`.* TO 'example_v10a'#'localhost';
Import an SQL file in to an existing database...
SOURCE C:\path1\path2\the.sql
Describe a database...
DESCRIBE table_name;
Change an SQL user's password...
SET PASSWORD FOR 'root'#'localhost' = PASSWORD('lulz_bad_password');
It's VERY important to realize in the long run that you'll be COMPOSING SQL queries, you NEVER dump them in a loop. Your goal is to make larger more elegant SQL queries that do more in fewer queries if not a single query whenever possible. In example a typical page request on my site's software utilizes about eight~12 queries subjective to the section/page/permissions/etc. Poorly written software like WordPress will utilize dozens if not more requests.
You will absolutely not be able to do anything effectively with databases without learning how to do RELATIONAL SQL; that is just fancy way of saying you know how to use INNER JOIN and LEFT JOIN (and some people use OUTER JOIN). A JOIN allows you to store a piece of information in the entire database once, like an account name. If you build a forum and want to pull forum posts you would JOIN the user accounts table and simply pull the user names by dynamically referencing them, you wouldn't actually store their user name every single time. However that is for another day. I learned SQL over a couple months and once I learned relational SQL I exploded making a blog in three weeks, a forum in a month, a chat room in three weeks, etc.
Feel free to use this as a cheat-sheet and give you some sense in how to reference some technical jargon. when you ask questions you need to be as concise and accurate with terminology as possible. Asking extremely large questions (how can I make a facebook without doing any work?) is bad, asking a specific question is good (How do I have my programming language connect to a database with a specific user that has limited privileges?)
The table names are variable, but what is certain is that SELECT only is allowed and certain tables are excluded (ie Users, Log). I'm making a reporting form where a user can just enter sql queries to make template reports.
SELECT 'field1' As 'foo', 'field2' as 'bar'.. 'fieldn'
FROM 'table1',..'tablen'
JOIN ... ON ...
WHERE CONDITION
Although I'm thinking I can have the table names in a html select list of existing tables.
Also make a user reporter_appname#localhost with SELECT access only to all tables except Users and Log? In that case I won't need to bother with a regex check of the query?
(This would be in PHP)
(Ideally I just wanted a single textarea where the admin can just type their query, my report function would then take the output and present it nicely etc.)
I suggest you re-think your design.
Identifying valid select statements (and excluding all other statements) is basically impossible without completely parsing SQL. A regex is not going to be up to the task.
Even if you allow only select statements, users could perform denial-of-service attacks on your database. It is very easy to create select statements that run forever (we've all done it). A malicious user could crash your site in a hurry. And even well-intentioned users might do this by accident.
It would be much better to give the users more limited options for creating reports. Let them select certain tables and columns from a list, and create the appropriate query for them.
There is probably free MySQL reporting software out there that could serve as a good starting point, though I don't have any experience with this myself.
I think that you should rethink the design of your application.
The Users and Log tables should be on one database and the tables with the data for the reports should be on another database.
If you have them all in one database already just create another database, link them and then create synonyms from one database to another only for the tables that the user can access via his queries.
The user will run his queries on the database you have just created and he will be limited to those tables that have synonyms on it.
I do not know if this would be the best option because your description of the case is relatively vague but based on the information I have this could be a solution.
We have redirect huge internet service from domain .de to domain .com - this is discussion board (vBulletin). At the moment we need to change all phrases like "domainame.de" to "domainame.com":
Over 120 tables (posts, threads)
A lot of MySQL fields
Anyone have suggestion how do something like this? We need replace string "domainame.de" to "domainname.com" - everywhere.
What you want to do sounds dangerous, as it could hit some false-positives, and change things unintentionally. Suppose your old domain is 'acme.de' and the new one 'acme.com', and some random visitor posted the following (this is an over-simplified example):
I enjoy working with Acme.Depending on my mood.
It would be very easy to convert this to:
I enjoy working with Acme.compending on my mood.
Therefore, my suggestions, in order of preference:
Don't update the DB at all, just configure your web server to redirect the .de traffic to the .com traffic. You're less likely to make mistakes this way.
If you must update the discussion board, do it in your display logic, rather than in the database--then you'll have no chance of making irrevocable mistakes.
Write a script in perl, or your favorite text-processing language, language, which does a regex replacement on every table/field. I suggest the following strategy:
a. Do a SELECT id,<field name> FROM <table> WHERE <field name> LIKE '%domain.de%'
b. Store output in a CSV, or other format that is easy to parse.
c. Run your Regex script to change domain.de to domain.com
d. Check the output.
e. Do an UPDATE <table> SET <field>=? WHERE id=?, with the output of your script.
Do it in small chunks at first (a few tens or hundreds of posts at a time), and visually check the results before committing your changes to the database, to make sure you haven't made mistakes.
I dont know whether this might work for you but have a look at this
The following query would give you the list of tables (120 tables)
SELECT DISTINCT table_name
FROM
(
SELECT *
FROM information_schema.`COLUMNS` C
) t1
WHERE TABLE_SCHEMA='schema_name'
next
you can use UPDATE query for each table. you can achieve this using CURSORS
I am not good in cursors but I think it will help in this situation