We are trying to export the logic to an Excel file. Reading through multiple levels of variables is quite a cumbersome process. Is there a piece of code that I can use to export the code to a text file?
Related
We are trying to import CSV file into Access Database via Powershell. My input file size is 1GB and it is getting difficult to iterate through each row and use insert command. Any quick suggestions here are highly appreciated
Thanks!!
as expresssed by #AlbertD.Kallal - what is the reason to use powershell at all? ... I simply made an assumption that you sought something that would run automatically, daily, unattended - - - as that is a typical reason.
if that is the case then it really breaks down to 2 parts:
make the import work manually in Access - - and then set up that import to fire automatically upon start/open of the Access file (auto exec).
just use powershell to start/open the Access file daily (or whenever...).
Access is not designed to be open full time and run unattended. So this is the typical approach to use it in that mode.
Ok, now having stated no need for power-shell, there are cases in which the IT folks and people are using power-shell to automate processes. So it not "bad" to consider power-shell - especially if it is being used.
I only wanted to point out that PowerShell will not help performance wise - and probably will be slower.
If you have (had) to say schedule a import to occur every 15 min or whatever?
Then I suggest setting up a VBA routine in a standard code module in Access to do the import. You then in the power shell, or windows script launch access, and then call that import routine. So, first step is to setup that routine in Access - even if using some kind of batch system for scheduling that import routine to run.
So, you use the windows scheduler.
It would: launch access, run the VBA sub, shutdown Access.
And using the windows scheduler is quite robust. So, we don't need (or want) to keep access running, but only launch it, run the import, and then shutdown access.
Next up, if the import process is "huge" or rather large, then on startup, a temp accDB file can be created, and then we import into that. We then can take the import table and send it into the production data table. (often column names are different etc. It also of course much safer to import into that temp table, and better yet, we can delete that temp file after - and thus we never suffer bloating or file size problems (no need to compact + repair).
So, the first thing to do is manually import the csv file using the Access UI. This ALSO allows you to create + setup a import spec. That import spec can thus remember the data types (currency, or often date time columns).
Once we have the import working and the import spec created?
Then we now can write code to do the above same steps, and THEN take the imported table and put that data into the production data table.
It it not clear if you "stage" the imported csv into that temp table, and then process that table into the real production data table, but I do suggest doing this.
(too dangerous to try and import directly into the production data).
you also don't share right now what kind of pre-processing, or what additonal code is required after you do the import of that csv (but, still, we assume now that such imports will be into a new temp table).
So, I would assume the steps are:
we import the csv file using built in import abiity of Acces
we then send this data table to the production table, perhaps with some code processing of each row before we send that temp table to production table.
Once we done the import, then we dump + delete the temp accDB file we used for the import, and thus we eliminate the huge data bloat issue.
Thus for the next time, then we create that temp file for a fresh import, and thus each time we start out with a nice empty database file.
So the first question, and you can create a blank new database for this test. Do you or can you import the csv file using Access. You want to do this, since such imports are VERY fast and VERY high speed. Even if the imported format is not 100% as you want, you do need to confirm and try if using the access UI you can import the csv file. if you can, then we can adopt VBA commands to do the same thing, but no use writing code if a simple csv import via Access UI can't be used.
I have millions of documents in different collections in my database. I need to export them to a csv onto my local storage when I specify the collection name.
I tried mlcp export but didn't work. We cannot use corb for this because of some issues.
I want the csv to be in such a format that if I try a mlcp import then I should be able to restore all docs just the way they were.
My first thought would be to use MLCP archive feature, and to not export to a CSV at all.
If you really want CSV, Corb2 would be my first thought. It provides CSV export functionality out of the box. It might be worth digging into why that didn't work for you.
DMSDK might work too, but involves writing code that handles the writing of CSV, which sounds cumbersome to me.
Last option that comes to mind would be Apache NiFi for which there are various MarkLogic Processors. It allows orchestration of data flow very generically. It could be rather overkill for your purpose though.
HTH!
ml-gradle has support for exporting documents and referencing a transform, which can convert each document to CSV - https://github.com/marklogic-community/ml-gradle/wiki/Exporting-data#exporting-data-to-csv .
Unless all of your documents are flat, you likely need some custom code to determine how to map a hierarchical document into a flat row. So a REST transform is a reasonable solution there.
You can also use a TDE template to project your documents into rows, and the /v1/rows endpoint can return results as CSV. That of course requires creating and loading a TDE template, and then waiting for the matching documents to be re-indexed.
I have tried the Cypher export csv, but the query itself takes too long, and it didn't seem practical since i needed to export a whole database which ranges upwards to a few gigabytes in size.
Is there a tool that lets me export a huge database in fairly good time?
You should take a look at the APOC procedures. This is straightforward to install within your Neo4j environment and provide many additional functionalities, e.g. to export your results or data to .csv file.
What is the difference between import a large csv file into SAS by code and using the import wizard (point and click)? Will the data imported in SAS be different?
The Import Data wizard is a code-writer, so it actually writes SAS code to perform the input. As such, it is no different than writing your own PROC IMPORT step, except that it has certain defaults.
You can check an option to ask it to write the code it produces to the log; that is often what I use the point and click interface for, to write the code partially, and then I update specifics it doesn't handle properly.
It depends. There are several ways to import a file:
Using GUI task in EG, SAS Studio or Base
PROC IMPORT
Data step code which manually specifies the fields and types.
The third is the most accurate because you've explicitly specified things.
If you do the same in the EG task your data will be the same. However, if you don't modify the automatic settings you'll get something more similar to PROC IMPORT.
PROC IMPORT guesses at types and lengths so it may not be correct.
Basically, there is no guarantee that the results of any of these methods will be the exact same depending on what options you pick.
The most accurate is the data step, followed by EG task where you customize each format/type and then PROC IMPORT with GUESSINGROWS=MAX option. This option forces SAS to scan the entire file before picking lengths and types so it's more likely to be accurate.
The difference is your comfort with either method. As #Joe answered, the Import Data wizard writes SAS code. Thus, it has all of the options available to it which are available in the PROC IMPORT.
The structure of the data imported, relative to its original structure in the CSV file solely depends on the options you chose to use in either import method.
Short Answer: The data will be the same.
I have a large CSV file (5.4GB) of data. It's a table with 6 columns a lot of rows. I want to import it into MySQL across several tables. Additionally I have to do some transformations to the data before import (e.g. parse a cell, and input the parts into several table values etc.). Now I can either do a script does a transformation and inserts a row at a time but it will take weeks to import the data. I know there is the LOAD DATA INFILE for MySQL but I am not sure how or if I can do the needed transformations in SQL.
Any advice how to proceed?
In my limited experience you won't want to use the Django ORM for something like this. It will be far too slow. I would write a Python script to operate on the CSV file, using Python's csv library. And then use the native MySQL facility LOAD DATA INFILE to load the data.
If the Python script to massage the CSV file is too slow you may consider writing that part in C or C++, assuming you can find a decent CSV library for those languages.