I can see that I can create/upload parquet files on on premise ssis with Azure Feature pack to my azure storage account.
However there is a lack of settings for parquet files, like partition or compression settings. The last one only available for text files.
what are my options to put a snappy compressed parquet file on my storage without databricks, just from my on premise ssis to storage account.
I tried it and you are correct, this is not supported yet.
You can share your feedback here so that the product group can look into this.
Alternately, here are some helpful docs you can pursue. Or perform compression on files in SSIS using the ZipFile Class in the recent version of the .NET framework.
Related
I am new to cloud foundry. I am currently working on a requirement where I have to upload a CSV file (via JSP UI) into a service deployed in cloud foundry and persists its data in service.
The issue is from UI, I only get a local path of that CSV file and when I am trying to parse that CSV via this path the file is not recognized. I guess the reason is service is already deployed in CF, so it does not recognize this local machine path.
Can you please let me know how can I parse this CSV file in local machine and where to parse this CSV.
Thanks in Advance!
There is nothing specific to Cloud Foundry about how you would receive an uploaded file in a web application. Since you mentioned using Java, I would suggest checking out this post.
How to upload files to server using JSP/Servlet?
The only thing you need to keep in mind that's specific to Cloud Foundry is that the filesystem in Cloud Foundry is ephemeral. It behaves like a normal filesystem and you can write files to it, however, the lifetime of the filesystem is equal to the lifetime of your application instance. That means restarts, restages or anything else that would cause the application container to be recreated will destroy the file system.
In short, you can use the file system for caching or temporary files but anything that you want to retain should be stored elsewhere.
https://docs.cloudfoundry.org/devguide/deploy-apps/prepare-to-deploy.html#filesystem
I have a problem with CKAN 2.0, just installed on a VirtualBox VM Ubuntu 12.04 64Bit.
I followed all steps of the official documentation but when I upload a file CSV into a dataset, the preview don't work because dataviewer use Dataproxy in place of Datastore.
My question is: CKAN 2.x uses only the dataproxy for prevew csv/xls files or I can use the datastore as in the previous versions? Reading here http://docs.ckan.org/en/ckan-2.0/data-viewer.html?highlight=datastore#viewing-structured-data-the-data-explorer is not clear if Datastore is used for all type of file or for all type except CSV/XLS.
Thank you! ;)
If you have the DataStore enabled and the data for that resource (data file) stored in the datastore then the datastore rather than the dataproxy will be used for the preview.
Think of the dataproxy as a fallback that kicks in if the DataStore is not enabled (or you have not stored that data file in the datastore).
I want to standardize and parameterize values across multiple environments and not have to change the dtsx files at any point.
The pattern I am deciding to use is to run all packages from the DTEXEC program and to specify configuration file on the command line and put that all in a batch file. with a different batch file for each environment.
One requirement is the location for the configuration file cannot be in the same physcial drive location, ie all config files are in D:\SSIS\config files. The main reason is that the production machine has an E drive mapped and this is where the ssis packages live and operate from. And, the staging machine does not, and cannot have a drive mapped to E.
Also, we want all files to reside in same pattern across all environments. config files in one place, archive files in another, etc. And, to try to use one medium, meaning the file system is where we store the packages, config files and batch files, as opposed to having data and artifacts in the registry and environment variables.
Does anyone see a more direct approach that satisfies all the conditions?
There may not be one and I thank you for your time...
That's how we're doing it - all config files on the file system, running packages using batch files that call dtexec, and passing config file locations to dtexec via parameters.
Watch out for a possible nasty gotcha, though. As this Books Online article points out, dtexec's behavior regarding command-line configurations changed between SSIS 2005 and SSIS 2008.
How do I set up a scheduled job in SQL Server 2008 that connects to a ftp server and transfers a file either to a local drive or even better into a table.
I think I can use SSIS (at least that I read) but can't find a tutorial or like.
If I can just get the file to my local drive then I good but of course the even better solution is a file transfer directly into a table :)
Definetely possible with SSIS.
Use Business Intelligence Studio to design the SSIS package, then you can schedule this package as SQL Agent job.
SSIS can't transfer it directly to the table, but it can download it to temporary file (use FTP task), then load it to table (depends on file format, most likely you'll need to use Data Flow task), and delete file (File System task).
Is there a way in which I can export my server settings from WAS (running under RAD 6) such that other developers will be able to use the same script to set up their environment?
To do this manually in RAD 6.x, simply right-click the server name in the "Server" view and choose one of:
Export server configuration to server
Import server configuration from server
The choice of wording here is potentially confusing. An import takes a configuration from the already-configured server and imports it into your workspace as a Configuration Archive (.car) file. An export asks for the location of a Configuration Archive (which must be in your workspace) and exports the settings it contains onto your server.
Yes, I agree that this sounds completely backwards.
Fortunately, the names are much more sensible in RAD 7.x. The options are:
Server configuration -> Backup...
Server configuration -> Restore...
These behave just as you'd imagine (Backup creates an archive file and Restore imports settings from an existing archive file.)
Important note: This process will not export service integration buses. However, I have had success including buses with the following steps:
Export a CAR file
Rename to .zip file for easy viewing
Manually copy the following files from your server profile into the archive:
cells/<cell_name>/buses/*
cells/<cell_name>/nodes/<node-name>/servers/server1/sib-engines.xml
Rename the archive back to .car
Note that this process is probably highly dependent on my specific configuration, but seems worth mentioning, since it has saved me a lot of trouble.
Another tip: Any files and folders you place inside the CAR will be dumbly copied into your profile directory whenever restoring a server configuration from that archive. This is convenient, because you can include necessary third-party libraries in the CAR file and reference them via WAS variables relative to your profile directory, resulting in one less thing for developers to download or configure.
You can export and import the profile with all its configuration using AdminTask export and import commands with the wsadmin scripting tool. If you are really serious also at the same time about how you release the applications to production environments you should probably create wsadmin scripts for deploying all your required settings in any case.
Also you might want to consider distributing virtual machines or simply copying the server installation otherwise from a reference installation.