I have an insanely large csv file which I want to import in postgres db. It is of the size of 500MB. I do not want to create the tabe first with more than 1000+ columns and then go for the insert like with the convntional copy command. Is there any way where I can use the header info of csv ( column names ), to directly import this data without creating a table first.
I am lookig for an import which is similar to R import.
It is probably not the solution your expecting but with Python you could read in the headers of your columns and create a table out of the csv very easily:
import pandas as pd
import psycopg2
from sqlalchemy import create_engine
# Read the csv to a dataframe
df = pd.read_csv('path_to_csv_file', index_col='name_of_index_column', sep=",")
# Connect and upload
engine = create_engine('postgresql+psycopg2://db_user_name:db_password#localhost:5432/' + 'db_name', client_encoding='utf8')
df.to_sql('table_name', engine, if_exists='replace', index =True, index_label='name_of_index_column')
Related
I am new to dynamo DB, I am trying to import (using Import from S3 in dashboard), a generic csv.
The csv has no unique id, so import gives error.
Is it possibile to import csv asking dynamo to generate id (and id column) when importing?
No, but you could preprocess the file(s).
I am New to Data Science, And I am using Spark with PySpark API. I want to Create a DataFrame of a .CSV File. And when i am doing that the Columns are moved to One Single Column as shown below.
The Commnds that i have used to Create a DataFrame of the CSV file are
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("MyFirstCSVLoad").getOrCreate()
df = spark.read.csv("order.csv")
df.Show()
Can anyone help me how to fix that issue.
The Link to the CSV File
https://mega.nz/file/opQFxQbJ#Csjk-CtAkb1CwB6F3hULk3xJxkAOdPyAMMCFjI30MEk
I look at your data and its seems that your delimiter is ";" and not comma.
In that case when you read your CSV file you should specifici the delimiter.
Use:
spark.read.option("delimiter", ";").csv(fileName)
I used MySQL workbench's import wizard to import data from a csv file. The import was successful but not all the data rows were imported. My csv file contains 271,117 rows but only 270,396 were imported successfully. 721 rows were not imported.
There was no error message at all but this is no good, right? How do I rectify this issue?
I had the same problem. But I converted my csv data to JSON using https://csvjson.com/
Afterwards I validated the JSON using https://jsonlint.com/.
Btw when you convert the data to JSON remember the first row needs to be the column names :)
Then inside of mySQL workbench select the table --> Table Data Import Wizard --> select the JSON file.
How to import only a single field in Robomongo while importing a JSON file in MongoDB?
I have tried but no queries of deleting the fields works.. so I want to import a single field. Is it possible?
i want import data like this E:\hai\tmp\r in the name_dir column
but went i import it its become like this e:haitmpr
how can i import it to db without losing backslash
please help me
im using sqlyog and import using csv file
Replace \ with \\ in your csv file then upload and see.
like- E:\\hai\\tmp\\r