Besides streaming a csv file yourself and painstakingly executing inserts for each line of data, is it possible to use the google cloud sdk to import an entire csv file in bulk, from inside a cloud function. I know in gcp console you can go to the import tab, select a file from storage and just import. But how can I emulate this programmatically?
in general, one has to parse the .csv and generate SQL from that; one line in the .csv would be represented by one INSERT statement. in the first place, you would have to upload the file - or pull it into Cloud Functions temporary storage, eg. with gcs.bucket.file(filePath).download.
then the most easy might be utilize a library, eg. csv2sql-lite - with the big downside, that one does not have full control over the import - while eg. csv-parse would provide a little more control (eg. checking for possible duplicates, skipping some columns, importing to different tables, whatever).
... and order to connect to Cloud SQL, see Connecting from Cloud Functions.
Related
I have an Excel file people use to edit data outside Azure Data Explorer (Kusto). What is the Kusto code I would use to ingest this data as needed into Kusto query?
So far it seems I need to use:
.create table (Name:type, Name:type)
to create a table.
If my CSV file is stored in OneDrive, what is the syntax to fill the table? Assume the file name is Sample.csv.
OneDrive location is not supported directly by Azure Data Explorer. However there are other options:
Using ingestion commands - you will need to place the files first in Azure Storage.
One Click Ingestion - is a feature of the Web Explorer tool, it will also can create the table for you. you can either download the files to your local computer or place it in Azure storage.
Using Import data from local file feature of Kusto Explorer (Windows client) (only works for local files)
I have created a SSIS package that reads from a CSV file (using the Flat file connection manager) and loads records into a database. I have deployed it on Azure data factory pipeline and I need to give the path of the CSV file as a parameter. I have created a azure storage account and uploaded the source file there as shown below.
Can I just give the URL of the source file for the Import file in the SSIS package settings as shown below? I tried it but it is currently throwing 2906 error. I am new to Azure - appreciate any help here.
First, you said Excel and then you said CSV. Those are two different formats. But since you mention the flat file connection manager, I'm going to assume you meant CSV. If not, let me know and I'll update my answer.
I think you will need to install the SSIS Feature Pack for Azure and use the Azure Storage Connection Manager. You can then use the Azure Blob Source in your data flow task (it supports CSV files). When you add the blob source, the GUI should help you create the new connection manager. There is a tutorial on MS SQL Tips that shows each step. It's a couple years old, but I don't think much has changed.
As a side thought, is there a reason you chose to use SSIS over native ADF V2? It does a nice job of copying data from blob storage to a database.
I'm trying to move a bunch of data that I have in a bucket (newline delimited json files) into BigQuery. BigQuery forbids certain characters in their field names, such as dashes - or slashes. Our data unfortunately has dashes in many of the field names, i.e.
jsonPayload.request.x-search
I tried renaming the field in the BigQuery schema to
jsonPayload.request.x_search hoping that the loader would do some magic, but nope.
Aside from running a job to rename the fields in storage (really undesirable, especially because new files come in hourly), is there a way to map fields in the JSON files to fields in the BQ schema?
I've been using the console UI but it makes no difference to me what interface to use with BQ.
I see a few option to work around this:
Create a Cloud Function to trigger when your new files arrive. Inside that function, read the contents of the file, and transform it. Write the results back to a new file and load it into BigQuery. I'm not sure how scalable this is in your situation. If your files are quite big, then this might not work.
Create a Cloud Function to trigger when your new files arrive, and then invoke a Dataflow templated pipeline to ingest, transform, and write the data to BigQuery. This is scalable, but comes with extra costs (Dataflow). However, it's a nice pattern for loading data from GCS into BigQuery.
Lazily, within BigQuery:
Import as CSV
One column per row, pick a delimiter that doesn't occur inside the files
Parse within BigQuery
Either with the BQ JSON functions
Or with javascript UDFs for maximum flexibility
At least this is what I usually do.
I have an idea on how to extract Table data to Cloud storage using Bq extract command but I would like rather like to know, if there are any options to extract a Big Query table as NewLine Delimited JSON to Local Machine?
I could extract Table data to GCS via CLI and also download JSON data from WEB UI but I am looking for solution using BQ CLI to download table data as JSON in Local machine?. I am wondering is that even possible?
You need to use Google Cloud Storage for your export job. Exporting data from BigQuery is explained here, check also the variants for different path syntaxes.
Then you can download the files from GCS to your local storage.
Gsutil tool can help you further to download the file from GCS to local machine.
You first need to export to GCS, then to transfer to local machine.
If you use the BQ Cli tool, then you can set output format to JSON, and you can redirect to a file. This way you can achieve some export locally, but it has certain other limits.
this exports the first 1000 line as JSON
bq --format=prettyjson query --n=1000 "SELECT * from publicdata:samples.shakespeare" > export.json
It's possible to extract data without using GCS, directly to your local machine, using BQ CLI.
Please see my other answer for details: BigQuery Table Data Export
I'd like to import CSV file into Salesforce automatically, e.g. once a day. I can't use Data Loader (Bulk API), because I don't have the needed edition. Is there any other, simple way?
Check to see if DBAmp (on the app exchange) is available for your edition. If it is, then you can use a SQL stored procedure to do it.
Otherwise, I think your only other option would be to schedule an Apex job in Salesforce to grab it from somewhere accessible (like an FTP).