Searched everywhere but couldn't find something similar to this.
I got a table as below (a glimpse of it):
customDimensions is the column that pulls the data in JSON format. However, there are some empty cells (as seen below) in the column.
I tried to Parse the whole column into JSON but it is returning me errors in the blank columns as below:
Therefore I tried an alternative way of first replacing those blank cells in the customDimensions to a generic JSON format input as {"Type":"Unhandled"} and then Parsed the column into JSON. This successfully got rid of the errors, but it is returning error while refreshing the data!
So, How do I handle this in a different way? I want to replace those blank cells as either "null" or any keyword, as long as I do not miss the count of those blank cells.
EDIT
Re: This successfully got rid of the errors, but it is returning error while refreshing the data! The error is as follows:
And that specific error is as above
EDIT 2:
Got trapped in the XY Problem but actually you can just replace the error after parsing the JSON instead:
Repeat with all columns and it'll be fine.
Related
I have come across one thing when we consider CSV as input to crawler
crawler doesn't identify the columns header when all the data is in string format in CSV.
#P1 Headers are displayed as col0,col1...colN.
#P2 And actual column names are considered as data.
#P3 Metadata (i.e. column datatype is shown as string even the CSV dataset consists of date/timestamp value)
If we are going to consider custom (CSV) classifier then we are manually mentioning the column header.
#P2 will get covered i.e. column names will be removed however
#P1 still remain same. column header will be displayed as col0,col1...colN.
There are 3 things I want to avoid and achieve expected result.
CSV with strings only should show actual column names instead of col0,col1...colN.
Metadata of generated table should show correctly (i.e. date/timestamp, string) once it is crawled by crawler.
If Custom classifier is used, we need to mention column header names manually in classifier, yet result is not satisfactory.
Need generic solution instead of manual interventions.
Have gone through this document: here
If anyone has already implemented the solution, Please help.
I got solution to one of the above points. Headers i.e. first line of CSV is displayed by using 'Has heading' in CSV classifier.
However, Solution for following is yet to figure out.
Metadata of CSV file is shown as string even if column contains timestamp/date value. Crawler is reading these datatypes as string.
Custom classifier needs manual interventions. I have mentioned all column names in classifier. Is there generic solution?
If we are using pd.to_csv to write the dataframe, then to avoid getting column names as col1, col2 and so on, add the parameter
index_label='index' such as:
pd.to_csv(df,index_label='index')
I have a flat file where rows have ids in form of guid. What I need is to redirect error output to table which will have error row id, error column and error code. The problem is that I can map only "Flat File Source Error Output Column" which is some sort of other columns concatenation. Is there a way I can get the ID column value of the error row? The best solution I could find is to add counter which will give the row number, but that's not exactly what I need, as ids are strings in my case.
Nope. You get 3* columns from the Flat File Source Component's: Flat File Source Error Output Column, ErrorCode, ErrorColumn.
A Source component defines the columns that all row buffers "downstream" of that point will contain. It is responsible for adding rows and then filling the columns in that new row buffer.
The Flat File Source component has a contract that describes how it should consume the source data - this many columns, this delimiter (or this many characters) etc.
What happens though when something overflows a length, or the data type is incompatible or not all of the delimiters are present? The design decision is to either put incomplete rows into the pipeline (but then how do you determine which columns get populated - fill left to right? what about type mismatch?) or treat it as an error. Normally, this blows up the data flow but if you add an Error output path, then you can see what row failed.
And the row is the atomic unit the flat file is using as input.
Read line -> Parse -> Write to Output [or Error] buffer {loop}
You could then use a Script Task to try and parse out the GUID from "Flat File Source Error Output Column" but then you have to hope that the value is in the row. It could be that a column has an embedded delimiter that wasn't escaped, someone transferred the file using the wrong encoding/line endings, etc.
*The Flat File Source Component does expose a property in the advanced editor, Component Properties tab for FileNameColumnName and that too will show up in the Error output path but that is the only source component I am aware of with this behaviour.
I am using the JMeter for the testing the application.
In the same, I have visited some page where the dynamic table contents should be displayed on the response data in view results.
However except that HTML dynamic table I am able to view the other contents so please help for getting the contents of the dynamic table too.
I have tried to get it using regular expression also have tried to get it with XPath extract. It goes till the <thead> tag, however, doesn't view the values in the <tbody>.
Here I have wanted to get the trains details. As the table is dynamic we should be able to get the values in the response of jmeter. However I am not able to get this contents.
In the same i tried to get the values of the special trains page using XPath so it gives me error of Illegal expression.
Screenshot no.3
I have not used any different library to get the html table data values.
I have tried to get the table contents by using the table it in regular expression
Xpath Query: "/html/body/table/tbody/tr2/td/div"
Reference name: Special_trains
so it gives the output java.net.URISyntaxException: Illegal character in query at index 77: http://enquiry.indianrail.gov.in/ntes/specialTrains.jsp?scrollerDivSpltrn=Not found
In the same the div id is: "scrollerDivSpltrn"
Hence need the answer
You just need to extract the table id properly. Once you are able to extract the that id, you should be able to fetch dynamic table.
I have a google chart and want to add a custom tooltip. I found some great answers like this this site and set about doing this with roles. I also found this link about it and it looked like the best way.
My data is being generated via json and I use a php file to create a json feed. The rows I have coded like this
{"cols": [ {"id":"","label":"Period","pattern":""},
{"id":"","label":"Recorded P/L","type":"number", "role":"data"} ,
{"id":"","label": null,"type":"string", "role":"tooltip"},
{"id":"","label":"Best Available P/L","type":"number", "role":"data"},
{"id":"","label": null,"type":"string", "role":"tooltip"}
]
Then it goes on and adds all the data. The problem is when I try to run this I get the error
All series on a given axis must be of the same data type
I have checked the json and that is formed correctly but am not sure what I could be doing wrong.
At least part of your problem is that you're not specifying the type for your first column.
I have started working with SSIS recently and I am hit a dead end with the look up transformation error rows redirect to a file. I have configured to send the rows that do not have a match to a flat file destination. But the file will not contain the collumn data, which is blank however the error code and error collumn into the text file.
This is the data contained in the text file:
,-1071607778,0
,-1071607778,0
,-1071607778,0
The first position suppose to be the data in my field but seems like its blank for a reason i don't understand. Any body help me clarify this, What am i missing???
that looks like the error number, not the row data