I am having real issues with a .db file its around 20gb in size with three tables and the rest data.
I am on a mac so i am having to use some crappy apps but it wont open in Access.
Does any one know what software will produce a .db file and what software will allow me to open it and export it as a CSV or MySQL file ?
Also if the connection was interrupted during transit could this effect the file ?
Since mac is BSD-based now, try opening a terminal and executing the command file /path/to/large/db -- it should tell you at least what file type the DB is, and from there you can determine what program to use to open it. It might be MySQL, might be PostGreSQL, might be SQLite -- file will tell you.
Example:
$ file a.db
a.db: SQLite 3.x database
$ file ~/.kde/share/apps/amarok/mysqle/amarok/tracks.{frm,MYD,MYI}
~/.kde/share/apps/amarok/mysqle/amarok/tracks.frm: MySQL table definition file Version 10
~/.kde/share/apps/amarok/mysqle/amarok/tracks.MYD: data
~/.kde/share/apps/amarok/mysqle/amarok/tracks.MYI: MySQL MISAM compressed data file Version 1
So it's SQLite v3? Then try
sqlite3 /path/to/db
and you can perform pretty much standard SQL from the CLI. At the CLI, you can type .tables to list all the tables in that DB. -- Or if you prefer a GUI, there are a few options listed in this question. Accepted answer was SQLite manager for Firefox.
Then you could drop tables or delete as you see fit.
Here's an example of dumping a csv to stdout:
$ sqlite3 -separator ',' -list a.db "SELECT * FROM t"
3,4
3,5
100,200
And to store it to a file -- the > operator redirects output to a file you name:
$ sqlite3 -separator ',' -list a.db "SELECT * FROM t" > a.csv
$ cat a.csv # puts the contents of a.csv on stdout
3,4
3,5
100,200
-separator ',' indicates that fields should be delimited by a comma; -list means to put row data on the same line, using the delimiter; a.db indicates which db to use; and "SELECT * FROM t" is just the SQL command to execute.
I'm not a Mac user but if it's a SQLite file I've heard great things about Base.
Related
I am working on a mysql (8) db which is too big for the 2TB linux partition size so I have moved the mysql instance onto a 16TB nvme raid under windows 10. All my other code is running on Debian 10 on a virtualbox instance and I have mapped a perm drive between the Debian VM and the nvme raid array.
I can open the database from debian and read and write as normal, so the odbc connector is working fine.
The issue here is the fact I am loading very large json log files into one table and doing it a row at a time was taking hours, so I opted to create a csv file for each log file and use LOAD DATA INFILE as part of the SQL statement.
Trouble is, when I execute the sql statement I get the 'file not found' issue, even though, looking at debug code, the file path is correct and it actually exists.
An excert from my python 3 code is:
p = f"/media/sf_unpack/{filename}"
try:
with open(p, 'r', buffering=1024 * 1024) as csvfile:
print(csvfile.read())
SQL = f"LOAD DATA INFILE '{p}' INTO TABLE xxx fields terminated by ',' LINES TERMINATED BY '\n' (field,field,.....etc) ;"
try:
mycursor.execute(SQL)
connection_object.commit()
except Exception as ex:
displayerror(ex)
This code will open the file correctly and show a value for p of /media/sf_unpack/filename.csv (which is correct).
When we get to the mycusror.execute(SQL) is raises an exception and says the directory or filename cannot be found. Interestingly, and I am sure this is the issue, the dubugger tells me the file that cannot be found is a windows version D:\media/sf_unpack/filename.csv - which looks as if it has something to do with the virtualbox mapping.
i.e. p =
I have tried to use the Path method from pathlib i.e p = Path(f"D:\mysql\unpack\{filename}" but that makes no difference.
I know I am doing something stupid but I am not sure what it is.
Any help would be gratefully recieved
I normally work with PHP/MySQL. A client wants to send variables from a .bat file - to a remote MySQL - where I will then manipulate them for display etc. I do not know how to connect and send these variables from a bat file in Windows.
I have small .bat file on windows, that simply writes a few variables to a text file.
#echo off
#echo Data: > test.txt
#echo VAR_1=777 >> test.txt
#echo VAR_2=245.67 >> test.txt
The result of the .bat file is a text file test.txt created with various details in it.
I would like the .bat file commands to also:
1) connect to a remote MySQL database
connect -> '8580922.hostedresource.com'
2) save to a basic table on a remote MySQL database:
INSERT INTO `My_Database`.`My_Table` (
`VAR_1` ,
`VAR_2` ,
)
VALUES (
'777',
'245.67'
);
Is this possible?
Is so - how?
I don't have MySQL Installed and I'm not familiar with it but here is a crack at something to try, based on info from the linked page.
REM This needs to be set to the right path
set bin=C:\Program Files\MySQL\MySQL Server 5.6\bin
REM set the host name and db
SET DBHOST=8580922.hostedresource.com
SET DBNAME=MyDatabase
REM set the variables and the SQL
SET VAR_1=777
SET VAR_2=245.67
SET SQL="INSERT INTO `My_Database`.`My_Table` (`VAR_1`,`VAR_2`) VALUES ( '%VAR_1%',
'%VAR_2%');"
"%bin%/mysql" -e %SQL% --user=NAME_OF_USER --password=PASSWORD -h %DBHOST% %DBNAME%
PAUSE
Please try that and post back the resulting error message. There are many reasons that it won't work, but you need to try it to find out.
I'm not sure where test.txt comes into this but it would be a good idea export the whole SQL statement to a text file then use the correct MySQL command line switch to just run the file instead of generating the SQL inside the batch file.
There's a bit more here.
connecting to MySQL from the command line
I am designing an application in Python and trying to write to a CSV file, but I am getting this error:
DatabaseError: 1 (HY000): Can't create/write to file '2015-04-06 20:48:33.418000'.csv (Errcode: 13 - Permission denied)
The Code:
def generate_report(self):
conn=mysql.connector.connect(user='root',password='',host='localhost',database='mydatabase')
exe2 = conn.cursor()
exe2.execute("""SELECT tbl_site.Site_name, State_Code, Country_Code,Street_Address, instrum_start_date, instrum_end_date, Comment INTO OUTFILE %s FIELDS TERMINATED BY '|' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\\\' LINES TERMINATED BY '\\n'FROM tbl_site JOIN tbl_site_monit_invent ON site_id = tbl_Site_site_id """, (str(datetime.datetime.now()),))
I can run this code without any errors on a Mac, but I need it to work on Windows.
How can I resolve this error?
Simple really. A colon character is not a valid character in a filename on Windows. It's not allowed.
Reference: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx
The colon character is in the list of "reserved characters", along with several others. (NOTE: One use of the colon character is as a separator for an Alternate Data Stream on NTFS. Ref: http://blogs.technet.com/b/askcore/archive/2013/03/24/alternate-data-streams-in-ntfs.aspx
Followup
The question has been significantly edited since my previous answer was provided. Some notes:
I'm not very familiar with running MySQL on Windows OS. Most of my work with MySQL server is on Linux.
The SELECT ... INTO OUTFILE statement will cause the MySQL server to attempt to write a file on the server host.
The MySQL user (the user logged in to MySQL) must have the FILE privilege in order to use the SELECT ... INTO OUTFILE statement.
Also, the OS account that is running MySQL server must have OS permissions to write a file to the specified directory, and the file to be written must not already exist. Also, the filename must conform to the naming rules for filenames on OS filesystem.
Ref: https://dev.mysql.com/doc/refman/5.5/en/select-into.html
For debugging this type of issue, I strongly recommend you echo out the actual SQL text that is going to be sent to the MySQL server. And then take that SQL text and run it from a different client, like the mysql command line client.
For debugging a privileges issues, you can use a much simpler statement. Test writing a file to a directory that is known to exist, that is known the mysql server has permissions to write files to, and with a filename that does not exist and that conforms to the rules for the OS and filesystem.
For example, on a normal Linux box, we could test with something like this:
mysql> SELECT 'bar' AS foo INTO OUTFILE '/tmp/mysql_foo.csv'
Before we run that, we can easily verify that the /tmp directory exists, that it is writable by the OS account that is running the mysql server, and that the filename conforms to the rules for the filesystem, and that the filename doesn't exist, e.g.
$ su - mysql
$ ls -l /tmp/mysql_foo.csv
$ echo "foo" >/tmp/mysql_foo.csv
$ cat /tmp/mysql_foo.csv
$ rm /tmp/mysql_foo.csv
$ ls -l /tmp/mysql_foo.csv
Once we get over that hurdle, we can move on to testing writing a file to a different directory, a file with a more more complex filename. Once we get that plumbing working, we can work on getting actual data, into a usable csv format.
The original question seems to indicate that the MySQL server is running on Windows OS, and it seems to indicate that the filename attempting to be written contains semicolon characters. Windows does not allow semicolon as part a filename.
It was simply permission error.
I am trying to run the standalone ncbi-blast-2.2.28+ on my machine (Mac) but get this error message when running blastp with SwissProt database:
BLAST Database error: Could not find volume or alias file (nr.00) referenced in alias file (/Users/me/bin/db/swissprot.00).
Here what I did:
1) downloaded the "ncbi-blast-2.2.28+-universal-macosx.tar.gz" from ncbi server and decompressed it
2) move the bin content of the folder to my $PATH directory "/Users/me/bin"
3) In "/Users/me/bin" I created a "db" folder, plus the ".ncbirc" file containing the following path:
[BLAST]
BLASTDB=/Users/me/bin/db
4) I downloaded the SwissProt database and got the following files in "/Users/me/bin/db/:
swissprot.00.msk
swissprot.01.msk
swissprot.02.msk
swissprot.03.msk
swissprot.04.msk
swissprot.05.msk
swissprot.06.msk
swissprot.07.msk
swissprot.08.msk
swissprot.09.msk
swissprot.10.msk
swissprot.00.pal
swissprot.01.pal
swissprot.02.pal
swissprot.03.pal
swissprot.04.pal
swissprot.05.pal
swissprot.06.pal
swissprot.07.pal
swissprot.08.pal
swissprot.09.pal
swissprot.10.pal
swissprot.pal
Then when I run blastp from any working directory (where my query file is), using this command:
blastp -query input.fasta -db swissprot
I get the following error message:
BLAST Database error: Could not find volume or alias file (nr.00) referenced in alias file (/Users/me/bin/db/swissprot.00).
As I read on other threads, I also tried to mention in the command line the whole path where the db is located, and to remove the .pal extension from the file names. But still doesn't work.
Can someone sees what I did wrong ?!!!!
you are storing your database files in db folder so you have to give this command instead of the one you have used:
blastp -query input.fasta -db db/swissprot
and I believe you are looking for an output in the console itself as you haven't used the -out option.
Also this will work only if the bin directory in which db folder is present be declared as an environment variable.
Have you checked the paths in .pal file?
Swissprot database that you have downloaded contains only links to entries in nr database: "nr - Non-redundant GenBank CDS translations + PDB + SwissProt + PIR + PRF, excluding those in env_nr". So you should additionally download nr database to run the standalone blast on your machine with SwissProt database. It weighs about 20 (!) Gb, but without it your blast will not work. Here's a link: ftp://ftp.ncbi.nlm.nih.gov/blast/db/
place all files from 00 to 10 folders into db and then check .pal file should contain 00 to 10 parts for example for nr databas its like
"nr.00" "nr.01" "nr.02" "nr.03" "nr.04" "nr.05" "nr.06" "nr.07" "nr.08" "nr.09" "nr.10"
I know you use the spool command when you are trying to write a report to a file in Oracle SQLplus.
What is the equivalent command in MySQL?
This is my code:
set termout off
spool ${DB_ADMIN_HOME}/data/Datareport.log # ${DB_ADMIN_HOME}/Scripts.Datavalidation/Datareportscript.sql
spool off
exit
How can I write it in MySQL?
In MySQL you need to use the commands tee & notee:
tee data.txt;
//SQL sentences...
notee;
teedata.txt == spooldata.txt
notee == spool off
For the Oracle SQLPlus spool command, there is no equivalent in the mysql command line client.
To get output from the mysql command line client saved to a file, you can have the operating system redirect the output to a file, rather than to the display.
In Unix, you use the > symbol on the command line. (It seems a bit redundant here to give an example of how to redirect output.)
date > /tmp/foo.txt
That > symbol is basically telling the shell to take what is written to the STDOUT handle and redirect that to the named file (overwriting the file if it exists) if you have privileges.
Q: is set pagesize and set linesize used in mysql when you are trying to generate a report?
A: No. Those are specific to Oracle SQLPlus. I don't know of any equivalent functionality in the mysql command line client. The mysql command line client has some powerful features when its run in interactive mode (e.g. pager and tee), but in non-interactive mode, it's an inadequate replacement for SQLPlus.
If I get what you are asking:
mysql dbname < ${DB_ADMIN_HOME}/Scripts.Datavalidation/Datareportscript.sql \
> ${DB_ADMIN_HOME}/data/Datareport.log
Use redirection.