I get CSV files of different length from different sources. The columns within the CSV are different with the only exception is each CSV file will always have an Id column which can be used to tie the records within different CSV files. At a time, two such CSV files needs to be processed. The process is to take the Id column from the first file and match the rows within the second CSV file and create a third file which contains contents from the first and second file. The id column can be repeated in the first file. Eg is given below. please note that the first file I might have 18 to 19 combination of different data columns so, I cannot hardcode the transformation within dataweave and there is a chance that a new file will be added every time as well. A dynamic approach is what I wanted to accomplish. So once written, the logic should work even if a new file is added. These files get pretty big as well.
The sample files are given below.
CSV1.csv
--------
id,col1,col2,col3,col4
1,dat1,data2,data3,data4
2,data5,data6,data6,data6
2,data9,data10,data11,data12
2,data13,data14,data15,data16
3,data17,data18,data19,data20
3,data21,data22,data23,data24
CSV2.csv
--------
id,obectId,resid,remarks
1,obj1,res1,rem1
2,obj2,res2,rem2
3,obj3,res3,rem3
Expected file output -CSV3.csv
---------------------
id,col1,col2,col3,col4,objectid,resid,remarks
1,dat1,data2,data3,data4,obj1,res1,rem1
2,data5,data6,data6,data6,obj2,res2,rem2
2,data9,data10,data11,data12,obj2,res2,rem2
2,data13,data14,data15,data16,obj2,res2,rem2
3,data17,data18,data19,data20,obj3,res3,rem3
3,data21,data22,data23,data24,obj3,res3,rem3
I was thinking to use pluck to get the column values for the first file. I idea was to get the columns in the transformation without hardcoding it. But I am getting some errors. After this I have the task of searching for the id and getting the value from the second file
{(
using(keys = payload pluck $$)
(
payload map
( (value, index) ->
{
(keys[index]) : value
}
)
)
)}
I am getting the following error when using pluck
Type mismatch for 'pluck' operator
found :array, :function
required :object, :function
I am thinking of using groupBy on id on the second file to facilitate better searching. But need suggestions on how to append the contents in one transformation to form the 3rd file.
Since you want to combine both CSVs without renaming the column names, you can try something like below
var file2Grouped=file2 groupBy ((item) -> item.id)
---
file1 map ((item) -> item ++ ((file2Grouped[item.id])[0] default {}) - 'id')
output
id,col1,col2,col3,col4,obectId,resid,remarks
1,dat1,data2,data3,data4,obj1,res1,rem1
2,data5,data6,data6,data6,obj2,res2,rem2
2,data9,data10,data11,data12,obj2,res2,rem2
2,data13,data14,data15,data16,obj2,res2,rem2
3,data17,data18,data19,data20,obj3,res3,rem3
3,data21,data22,data23,data24,obj3,res3,rem3
Working expression is as given below. The removing the id should happen before the default
var file2Grouped=file2 groupBy ((item) -> item.id)
---
file1 map ((item) -> item ++ ((file2Grouped[item.id])[0] - 'id' default {}))
Here is what I use.
OS: Linux Mint 18
Editor: LibreOffice Writer 5.1.6.2
Situation
Consider the following foo.csv file (just example, the raw data contains hundred of lines):
A,B,C
1,2,3
To create a table in Writer with the data from foo.csv usually one creates the table via Toolbar and then type the contents (possibly using TAB to navigate between cells).
Here is the result of procedure above:
Goal: Since the whole foo.csv contains hundreds of lines, how to proceed?
1st try: copy and paste the data from foo.csv into the table does not work, as seen below.
2nd try: copy and paste the data from foo.csv into the table with all cells selected does not work, as seen below.
Question: is it possible to edit an odt file in some way to write some code (like we could do with tags in HTML) to produce such table?
Embed a Calc spreadsheet is not acceptable.
Just use the "Text to Table" feature:
Insert the csv as "plain text" into your writer document (not into a table, just anywhere else);
Select the inserted lines;
Select Menu "Table" -> "Convert" -> "Text to Table";
Adjust the conversion properties as needed (set separator to comma: Select "Other", enter a comma into the box at the right"):
Hit OK - LO Writer will convert the text content of your CSV into a nice Writer table.
Please note that if you use this solution, there's nothing like a "connection" between the writer table and the csv data. Changing the csv won't affect the writer table. This would be possible only by embedding an object (but this won't result into a Writer table...).
If the csv data is the only content of the odt (writer) file, there's another option: Use LibreOffice Base to create a LO Database using the csv file (dynamically updated if the csv changes), and use the Report feature to get a tabular output of the csv data. LO Base will store the output layout as report, making it easy to create an up-to-date report.
I have a csv file that looks the following way:
I want to create a database from it in Neo4j. Rows are nodes with labels gene, columns are also nodes with labels cell. I need to write a CREATE query that would create all my gene and cell - nodes and a relationship one for each combination of gene and cell. Currently I am stuck with the following code:
LOAD CSV WITH HEADERS FROM 'file:///merged_full.csv' AS line
CREATE (:Gene {id: line.gene_ids, name: line.wikigene_name})
I need to somehow iterate over all columns - starting from index 3 - after creating gene nodes, but I do not know how to do that.
Here are 3 queries that, performed in order, should do what you want.
This query creates a temporary Headers node with a names property that contains the collection of headers from the CSV file. It uses LIMIT 1 to only process the first row of the file. It also creates all the Cell nodes, each with it own name property.
LOAD CSV FROM 'file:///merged_full.csv' AS line
MERGE (h:Headers)
SET h.names = line
WITH line
LIMIT 1
UNWIND line[3..] AS name
MERGE (c:Cell {name: name})
This query uses the APOC function apoc.map.fromNodes to generate a map named cells, which maps each cell name to its cell node. It also gets the Headers node. It then loads the non-header data from the CSV file (using SKIP 1 to skip over the header row), and processes each row as follows. It uses MERGE to get/create a Gene node, g, with the desired id and name. It uses the REDUCE function to generate a collection of the Cell nodes that have a "1" column value in the current row, and the FOREACH clause then creates a (g)-[:HAS]->(x) relationship (if necessary) for every cell, x, in that collection.
WITH apoc.map.fromNodes('Cell', 'name') AS cells
MATCH (h:Headers)
LOAD CSV FROM 'file:///merged_full.csv' AS line
WITH h, cells, line
SKIP 1
MERGE (g:Gene {id: line[1], name: line[2]})
FOREACH(
x IN REDUCE(s = [], i IN RANGE(3, SIZE(line)-1) |
CASE line[i] WHEN "1" THEN s + cells[h.names[i]] ELSE s END) |
MERGE (g)-[:HAS]->(x))
This query just deletes the temporary Headers node (if you wish):
MATCH (h:Headers)
DELETE h;
If the columns correspond with cell nodes, then you should know all the cell nodes you need just be looking at the CSV header.
I'd recommend writing a small query just to create each of the cell nodes you need, then create an index or unique constraint on :Cell(id) (or name, or whatever the property is that is meant to identify a :Cell).
At that point the problem becomes getting and processing each relevant column (I assume only the ones with 1 as the value). APOC Procedures may help here.
apoc.map.sortedProperties() can be used to take your line map and give you a list of key/value list pairs, which you can filter down to those where the key begins with 'V', and where the value is 1, then use what's remaining to match on the relevant :Cell node and create the relationship.
Could some one please guide me how to do this
I would like to import Test Data from the csv file into Vugen Scripts.
Lets say I have Payee.csv file which has 28 columns of Data and csv file located some shared location. In script I need to pass the values from the csv file by importing.How can I do that in Vugen script?
I know in QTP we have DataTable.Import.
Thanks,
Raj
You can import CSV file and use it as a data file for parameterization.
Follow the steps as mentioned below:
Select the value you want to parameterize. for an explanation, I took username field (jojo).
Right click on the value (jojo) -> select Replace With Parameter -> select Create New Parameter
Write Parameter name as username (any name of your choice) -> select parameter type as File -> Click OK button.
Select Yes/No for "Do You want to replace all occurrences" based on your need.
The value (jojo) will be replaced as {Username}. Username.dat file will be created with only one entry in the file i.e., jojo
Right click on {Username} -> select Parameter Properties -> click Browse button under File Type -> Select the CSV file (Users.csv) you want to import (select Save as type -> All Files (.*) in Save As dialogue box).
Parameter Properties dailougue box shows all the entries in the csv file in tabular format.
under Select Column, select By name radio button -> from dropdown, select "username" (column header) -> click Close
Repeat the actions for each column in the CSV file, wherever you want to replace the values with data in CSV (column header is the key).
Second iteration:
For password parameter, I followed the same process, given the name parameter name as "Password", and selected the same CSV file, under refer the data "By name" -> Password (column header)
Screenshot for Password parameter:
References:
http://lrhelp.saas.hp.com/en/latest/help/WebHelp/Content/Tutorial/Preparing.htm#How2
I often use Sublime Text 2's search all open files/folders feature (CMD+SHIFT+F).
Is it possible to perform a second search that only queries files which contain the first search query?
For example: Search 1 finds 5 files that contain "hello". Search 2 should only query those same 5 files for "world", so the resulting matches will be files which contain both "hello" and "world".
I can search the text which is visible in the results buffer (such as in the screenshot below), but I'd like to search the entire contents of all files that are shown in the results buffer.
1:
Open the find_in_files panel with Ctrl + Shift + F.
In the example, <open folders> is used for the Where: argument, but you can use whatever fits your initial search criteria.
2:
Open the find panel with Ctrl + F while you are still in the results buffer.
Enter ^(/Users/.*)(?=:$) as your Find: argument, and make sure the RegEx switch is enabled. This RegEx pattern will match all of the paths within the results.
( Windows users can enter ^(C:\\.*)(?=:$) )
Copy the selected paths.
3:
Paste the copied paths into a blank document.
Open the replace panel with Ctrl + H.
Enter ^(/Users/.*)(\n) as the Find What: argument, and $1$2, as the Replace With: argument. This will effectively convert all of the lines into a single line of comma separated values.
( Windows users can enter ^(C:\\.*)(\n) )
Copy the resulting line.
4:
Open the find_in_files panel with Ctrl + Shift + F.
Paste the copied path list as your Where: argument, and enter your second query as the Find: argument.
5:
Repeat as necessary.