To comply to regulations, I'm trying to download the purchase invoice documents (as PDF files) from some of my divisions to save them on-disk for archiving purposes.
I use Invantive Query Tool to do this. I like to know which table to use and how to export these attachments only regarding purchase invoice documents.
You can indeed do this by using the export options in Invantive Query Tool or Invantive Data Hub.
What you need is a query that hooks up the document information of type 20 (purchase invoices) with the actual attachment files. You can find a list of types and their description in the DocumentTypes view. You can find the document attachment files in the DocumentAttachmentFiles table.
When you have retrieved that, you can export the documents from that query to disk using a local export documents statement.
The full query is here:
use 123456
select /*+ join_set(dae, document, 10000) */ attachmentfromurl
, dct.division || '/' || dae.id || '-' || filename
filepath
from exactonlinerest..documents dct
join DocumentAttachmentFiles dae
on dae.division = dct.division
and dae.document = dct.id
where dct.Type = 20
order
by dct.division
, dae.id
local export documents in attachmentfromurl to "c:\temp\docs" filename column Filepath
Make sure to set the ID of the division right in the use statement (this is the technical ID, not the 'division number', which can contain duplicates). You can find that in the top menu bar under Partitions. Or simply use use all to get the documents from all divisions (this might take a while).
Also set the file path right where it says c:\temp\docs now. Then hit F5 in the Query Tool to execute, or run the script from Data Hub.
Related
I am trying to accomplish something that is pretty easy to do in SQL, but seemingly very challenging to do in SSIS without using SQL. Basically, I need to consolidate and concatenate a field of a many-to-one relationship.
Given entities: [Contract Item] (many) to (one) [Account]
There is a field [ari_productsummary] that contains the product listed on the Contract Item entity. We want to write that value to the Account as [ari_activecontractitems]. However, an Account may have more than one Contract Item record associated to it, in which case, we want to concatenate those values. We also only want the distinct values to be concatenated (distinct rows already solved within my data flow).
This can be accomplished by writing to a temporary table, and then using a query or view to obtain the summarized results as followed. I created a SQL table called TESTTABLE that contains the [ari_productsummary] from the Contract Item entity along with the referring [accountid] to map it back to Account. I then wrote the following query as a view:
SELECT distinct accountid,
(SELECT TT2.ari_productsummary + '; '
FROM TESTTABLE TT2
WHERE TT2.accountid = TT.accountid
FOR XML PATH ('')
) AS 'ari_activecontractitems'
FROM TESTTABLE TT
Executing that Query provides me the results that I want, which I can then use for importing into the Account entity as shown below:
But how do I do this in a SSIS dataflow without writing to a SQL table as a temporary placeholder for the data?? I want to do the entire process inside one dataflow container, without using a temporary SQL table/view. The whole summarization process needs to be done on the fly:
Does anyone have a solution that doesn't require a temporary SQL table/view/query, but is contained entirely within the data flow?
I am using VS 2017 and the KingswaySoft Dynamic CRM 365 ETL toolset to develop my solution/package.
Spit balling here as I don't Dynamics nor do I have the custom components.
Data Flow 1 - Contract aggregation
The purpose of this data flow is to replicate your logic in the elegant query you provided and shove that into a Cache Connection Manager (see Notes for 2008+ at the end)
KingswaySoft Dynamics Source -> Script Task -> Cache Transform
If you want to keep the sort in there, do it before the script task. The implementation I'll take with the Script Task is that it's fully blocking - that is all the rows must arrive before it can send any on. Tasks like the Merge Join are only partially blocking because the requirement of sorted data means that once you no longer have a match for the current item, you can send it on down the pipeline.
The Script Task is going to be asynchronous transformation. You'll have two output columns, your key accountid and your new derived column of ari_activecontractitems. That column will might need to be big - you'll know your data best but if it's a blob type in Dynamics (> 4k unicode or > 8k ascii characters) then you'll have to define the data type as DT_TEXT/DT_NTEXT
As inputs, you'll select accountid and ari_productsummary from your source.
The code should be pretty easy. We're going to accumulate the inbound data into a Dictionary.
// member variable
Dictionary<string, List<string>> accumulator;
The PreProcess method, we'll tack this in there to initialize our variable
// initialize in PreProcess method
accumulator = new Dictionary<string, List<string>>();
In the OnBufferRowSent (name approx)
// simulate the inbound queue
// row_id would be something like Rows.row_id
if (!accumulator.ContainsKey(row_id))
{
// Create an empty dictionary for our list
accumulator.Add(row_id, new List<string>());
}
// add it if we don't have it
if (!accumulator[row_id].Contains(invoice))
{
accumulator[row_id].Add(invoice);
}
Once you get the signal sent of no more data available, that's when you start buffering output data. The auto generated code will have placeholders for all this.
// This is how we shove data out the pipe
foreach(var kvp in accumulator)
{
// approximately thus
OutputBuffer1.AddRow();
OutputBuffer1.row_id = kvp.Key;
OutputBuffer1.ari_productsummary = string.Join("; ", kvp.Value);
}
We have an upcoming release that comes with a component that does exactly what you are trying to achieve without the need of writing custom code. The feature is currently under preview, please reach out to us for private access to the feature. You can find our contact information on our website.
UPDATE - June 5, 2020, we have made the components available for public access at https://www.kingswaysoft.com/products/ssis-productivity-pack/ as a result of our 2020 Release Wave 1. We have two components available that serve this kind of purpose. The Composition component will take input values and transform into a composite value in a SSIS column. The Decomposition component does the opposite, it would take an input value and split it into multiple rows using either delimiter-based text splitting or XML/JSON array splitting.
I need to upload some 2000 documents to specific users in salesforce. I have a csv file that has the Salesforce-assigned ContactID, as well as a direct path to the files on my desktop. Each contact's specific file url has been included in the csv. How can I upload them all at one and, especially, to the correct contact?
You indicated in the comments / chat that you want it as "Files".
The "Files" object is bit more complex than Attachments, you'll need to do it in 2-3 steps. What you see as a File (you might see it referred to in documentation as Chatter Files or Salesforce Content) is actually several tables. There's
ContentDocument which can be kind of a file header (title, description, language, tags, linkage to many other areas in SF - because it can be standalone, it can be uploaded to certain SF Content Library, it can be linked to Accounts, Contacts, $_GOD knows what else)
ContentVersion which is well, actual payload. Only most recent version is displayed out of the box but if you really want you can go back in time
and more
The crap part is that you can't insert ContentDocument directly (there's no create() call in the list of operations) .
Theory
So you'll need:
Insert ContentVersion (v1 will automatically create for you parent ContentDocuments... it does sound bit ass-backwards but it works). After this is done you'll have a bunch of standalone documents loaded but not linked to any Contacts
Learn the Ids of their parent ContentDocuments
Insert ContentDocumentLink records that will connect Contacts and their PDFs
Practice
This is my C:\stacktest folder. It contains some SF cheat sheet PDFs.
Here's my file for part 1 of the load
Title PathOnClient VersionData
"Lightning Components CheatSheet" "C:\stacktest\SF_LightningComponents_cheatsheet_web.pdf" "C:\stacktest\SF_LightningComponents_cheatsheet_web.pdf"
"Process Automation CheatSheet" "C:\stacktest\SF_Process_Automation_cheatsheet_web.pdf" "C:\stacktest\SF_Process_Automation_cheatsheet_web.pdf"
"Admin CheatSheet" "C:\stacktest\SF_S1-Admin_cheatsheet_web.pdf" "C:\stacktest\SF_S1-Admin_cheatsheet_web.pdf"
"S1 CheatSheet" "C:\stacktest\SF_S1-Developer_cheatsheet_web.pdf" "C:\stacktest\SF_S1-Developer_cheatsheet_web.pdf"
Fire Data Loader, select Insert, select showing all Salesforce objects. Find ContentVersion. Load should be straightforward (if you're hitting memory issues set batch size to something low, even 1 record at a time if really needed).
You'll get back a "success file", it's useless. We don't need the Ids of generated content versions, we need their parents... Fire "Export" in Data Loader, pick all objects again, pick ContentDocument. Use query similar to this:
Select Id, Title, FileType, FileExtension
FROM ContentDocument
WHERE CreatedDate = TODAY AND CreatedBy.FirstName = 'Ethan'
You should see something like this:
"ID","TITLE","FILETYPE","FILEEXTENSION"
"0690g0000048G2MAAU","Lightning Components CheatSheet","PDF","pdf"
"0690g0000048G2NAAU","Process Automation CheatSheet","PDF","pdf"
"0690g0000048G2OAAU","Admin CheatSheet","PDF","pdf"
"0690g0000048G2PAAU","S1 CheatSheet","PDF","pdf"
Use Excel and magic of VLOOKUP or other things like that to link them back by title to Contacts. You wrote you already have a file with Contact Ids and titles so there's hope... Create a file like that:
ContentDocumentId LinkedEntityId ShareType Visibility
0690g0000048G2MAAU 0037000000TWREI V InternalUsers
0690g0000048G2NAAU 0030g000027rQ3z V InternalUsers
0690g0000048G2OAAU 0030g000027rQ3a V InternalUsers
0690g0000048G2PAAU 0030g000027rPz4 V InternalUsers
1st column is the file Id, then contact Id, then some black magic you can read about & change if needed in ContentDocumentLink docs.
Load it as insert to (again, show all objects) ContentDocumentLink.
Woohoo! Beer time.
Your CSV should contain following fields :
- ParentID = Id of object you want to link the attachment to (the ID of the contact)
- Name = name of the file
- ContentType = extension(.xls or .pdf or ...)
- OwnerId = if empty I believe it takes your user as owner
- body = the location on your machine of the file (for instance: C:\SFDC\Files\test.pdf
Use this csv to insert the records (via data loader) into the Attachment object.
You will then see for each contact, that records have been added to the 'Notes & Attachments' related list.
I frequently need to pull some CSV reports and analyze them using powerpivot. The "issue" is that the tool spits out the report like this:
Report Name Keywords (Group contains 778600, Campaign contains us-en)
Client XYZ
Scope Entire Account
Date Range 3/12/2015
Filters Campaign contains us-en; Group contains 778600; Clicks > 0; Reduced Dimension
Keyword Account Publisher Campaign Group Search Bid $ Status Destination URL
Total for all 2 keywords
Keyword Account Publisher Campaign Group Search Bid $ Status Destination URL
bla bla bla Account Name Publisher Name Campaign Name Group Name 1 Active URL
So what i always need to do is to remove the first 9 rows of the CSV prior to importing. Usually i can do this on Notepad++, but sometimes the CSV is so large that i actually can't really open it to edit. So far i'm using a program called 010 Editor, but i have only some days left of it.
Is there an easy way to skip those rows when importing?
Thanks a lot
You can use Power Query (free to download) to load data to Power Pivot. It allows you to skip the first x rows and filter out rows with blank/null values. Once you are able to get this to work once, you can copy the M code to use it on other CSVs. Or you can automate it as a function and just feed it file locations.
I am running a classic vbscript ASP site with a SQL2008 database. There are a few pages that are processor-heavy, but don't actually change that often. Ideally, I would like the server to process these once a night, perhaps into HTML pages, that can then fly off the server, rather than having to be processed for each user.
Any ideas how I can make this happen?
The application itself works very well, so I am not keen to rewrite the whole thing in another scripting language, even if classic asp is a bit over the hill!!
Yes :
You didn't specify which parts of the pages are "processor heavy" but I will assume it's the query and processing of the SQL data. One idea is to retrieve the data and store it as a cached file, in the filesystem. XML is a good choice for the data format.
Whereas your original code is something like this:
(psuedocode)
get results from database
process results to generate html file
...your modified code can look like this:
check if cache file exists
if not exist
get results from database
store results in cache file
get results from cache file
process results to generate html file.
This is a general caching approach and can be applied to a situation where you've got
query parameters determining the output. Simply generate the name of the cache file based on all the constituent parameters. So if the results depend on query parameters named p1 and p2, then when p1 and p2 have the values 1234 and blue respectively, the cache file might be named cache-1234-blue.xml . If you have 5 distinct queries, you can cache them as query1-1234-blue.xml, query2-1234-blue.xml and so on.
You need not do this "nightly". You can include in your code a cache lifetime, and in place of the "if cache file exists" test, use "if cache file exists and is fresh". To do that just get the last modified timestamp on the cache file, and see if it is older than your cache lifetime.
Function FileOlderThan(fname, age)
'function returns True if the file is older than the age,
' specified in minutes.
Dim LastModified, FSO, DateDifference
Set FSO = CreateObject("Scripting.FileSystemObject")
LastModified = FSO.GetFile(fname).DateLastModified
DateDifference = DateDiff("n", LastModified, Now())
If DateDifference > age Then
FileAge = False
Else
FileAge = True
End If
End Function
fname = Server.MapPath(".") & cacheFileName
If FileOlderThan(fname, 10) Then
... retrieve fresh data ...
End If
This could be 10 minutes, 10 hours, 10 requests, whatever you like.
I said above that XML is a good choice for the data format in the cachefile. ADO has a SaveAsXML method, and you can also generate XML directly from SQL2008 using the FOR XML clause appended to the query.
If the "processor heavy" part is not the query and retrieval, but is the generation of the html page, then you could apply the same sort of approach, but simply cache the html file directly.
I have tree tables, Customer, Invoice and InvoiceRow with the standard relations.
These I have to export in one fixed field length file with the first two characters of each row identifying the row type. The row types have different specifications.
I could probably do it with a nested loop in a script block, but this is my first ever SSIS package and that solution feels wrong.
edit:
The output has to have:
Customer
Invoice
Rows
Customer
Invoice
Rows
and so on
Your gut feeling on doing this using a Script Destination component is correct. Unfortunately, this scenario doesn't jive with SSIS well. I don't consider this a beginner package. If you must use SSIS then I'd start by inner joining all the data so there is one row for each InvoiceRow, containing the data needed from all three tables.
CustomerCols, InvoiceCols, RowCols
Then, in the script destination component you'll need to keep track of the customer and invoice values, as they change you'll need to write extra rows to the output.
See Creating a Destination with the Script Component for more information on script destination.
My experience shows that script destinations can have good performance.
I would avoid writing Script Destination, and use just Script Transform + Flat File Destination. This way, you concentrate on the logical output (strings of data), while allowing SSIS to do actual writing to the file (it might be a bit more efficient, plus you concentrate on your business, not on writing to files).
First, you'll need to get denormalized data. You can do joins and sorts in the DBMS, but if you don't want to put too much pressure on DBMS - just get sorted data out of it and merge it using two SSIS Merge Join transforms.
Then do the script: keep running values of current Customer and Invoice, output them when they change, output InvoiceRow on every input. Something like this:
if (this.CustomerID != InputBuffer.CustomerID) {
this.CustomerID = InputBuffer.CustomerID;
OutputBuffer.AddRow();
OutputBuffer.OutputColumn = "Customer: " + InputBuffer.CustomerID + " " + InputBuffer.CustomerName;
}
// repeat the same code for Invoice
OutputBuffer.AddRow();
OutputBuffer.OutputColumn = "InvoiceRow: " + InputBuffer.InvoiceRowPrice;
Finally, add a Flat File Destination with a single column (OutputColumn created by the script) to write this to the file.
Process your three tables so that the outputs are all appropriate for your output file (including the row type designator). You'll have to do this in three separate flow paths in your data flow, then bring the rows together in a Union All data flow element. From there, process them as needed to create your output file.