I am able to send POST/PUT requests, but I am not sure what I need to send for the SODA API to automatically generate a row identifier(id) for each row. The content type is "text/csv".
To provide a quick example, I can upload a file for my dataset on the Socrata website and the IDs are automatically generated-I am able to access "https://socrata-dataportal.org/.../1.json". By default, 1 row identifier is given for each row, from 1 to however many rows there exist.
I can upload a file through manual HTTP requests, and the data is posted correctly, but there is no id generated for them, i.e, going to the same page "https://socrata-dataportal.org/.../1.json" will give me a "row.missing" error code.
The internally assigned row identifiers are not always guaranteed to be created in numeric order. To get a list of the internally assigned row identifiers you need to issue this GET request on the dataset, for example:
https://socrata-dataportal.org/resource/abcd-1234.json?$select=:id
This will output all internally assigned row identifiers for the dataset identified by 'abcd-1234'.
Related
I am trying to upload a csv file from Google Cloud Storage (GCS) to BigQuery (BQ) and auto-detect schema.
What I tried to do is enable auto-detect schema and enter the number of rows to skip in "Header rows to skip" option. I have 6 rows which contain descriptive information about the data which I need to skip. The 7th row is my actual header row.
According to Google's documentation in: https://cloud.google.com/bigquery/docs/schema-detect#auto-detect:
"The field types are based on the rows having the most fields. Therefore, auto-detection should work as expected as long as there is at least one row of data that has values in every column/field."
The problem with my CSV is that the above condition is not met in the sense that I have nulls in the rows.
Also, my CSV contains many rows which do not include any numerical values which I think adds an extra complexity for Google's schema auto detection.
The auto detect is not detecting the correct column names nor the correct field types. All field types are being detected as strings and column names assigned as such: string_field_0 , string_field_1, string_field_3 ,...etc. It is also passing the column names of my CSV as a row of data.
I would like to know what I can do to correctly upload this CSV to BQ with skipping the leading unwanted rows and having the correct schema (field names and field types).
You can try using tools like bigquery-schema-generator to generate the schema from your csv file and then use it in a bq load job for example.
After reading some of the documentation, specifically the CVS header section I think what you're observing is the expected behavior.
An alternative would be to manually specify the schema for the data.
Solved this by including my actual header row in the csv in the number of rows to skip.
I had 6 rows I actually needed to skip. The 7th row was my header (column names). I was entering 6 in the Header rows to skip.
When I entered 7 instead of 6, the schema was auto detected correctly.
Also, I realized that in this sentence in Google's documentation: "The field types are based on the rows having the most fields. Therefore, auto-detection should work as expected as long as there is at least one row of data that has values in every column/field.", nulls are considered values and so that was not actually causing a problem in the upload to BQ.
Hope this helps someone facing the same issue!
logged into the website by using the csv .By default particular customers data will be displayed on the page. Different customers would be present in the dropdown where the Admin has the access to select the customer and then the data of that particular customer would be displayed on the screen
Used blazemeter to generate the jmeter script where the default data response is generated all the time instead of the selected customer ID?
Kindly help me know how to pull the latest selected customer data In the jmeter and use that customer data in the consecutive steps?
if you mean you want to dynamically select customer and fetch data respective to the selective customer and want to use that data in the consecutive steps.
For this
First identify from which endpoints your frontend is receviving list of customers and their details.
Once identified you will be having 2 endpoints
1st which provide all the list of customer/ or select is pre-rendred
2nd which provide detail for that customer
Now extract data from 1st endpoint or HTML if pre-renderd save it in varaibe
Now call the 2nd endpoint and extract the required data and save it in variable
now the variable can be used in the consecutive requests
JMeter doesn't know anything about "dropdowns", it acts on HTTP protocol level so by default it will always execute the HTTP Requests which you have recorded.
So when you "select" the customer it generates some specific value for the HTTP Request, you can check the values for the different customers using browser developer tools, put them into a CSV file and replace the recorded hard-coded customer name or ID with the JMeter Variable from the CSV file.
With regards to your question itself, the exact solution will depend on the dropdown implementation, it's hard to suggest anything meaningful without seeing the HTML code for the dropdown.
Here is an example of how you can get the last value for the HTML Select dropdown using XPath Extractor:
I am working to create a database. I have excel spreadsheets that will be uploaded which have two rows of titles: the first being the parameter and the second being units. I am new to access and all I know about uploading into it is using the import>excel command. When I try to upload one of these spreadsheets, access crashes (I assume because it's trying to interpret different types of data in a single column). How do I designate this second row as units and not data values?
I have a file like as seen below: Just Ex:
kwqif h;wehf uhfeqi f ef
fekjfnkenfekfh ijferihfq eiuh qfe iwhuq fbweq
fjqlbflkjqfh iufhquwhfe hued liuwfe
jewbkfb flkeb l jdqj jvfqjwv yjwfvjyvdfe
enjkfne khef kurehf2 kuh fkuwh lwefglu
gjghjgyuhhh jhkvv vytvgyvyv vygvyvv
gldw nbb ouyyu buyuy bjbuy
ID Name Address
1 Andrew UK
2 John US
3 Kate AUS
I want to dynamically skip header information and load flatfile to DB
Like below:
ID Name Address
1 Andrew UK
2 John US
3 Kate AUS
The header information may vary (not fixed no. of rows) from file to file.
Any help..Thanks in advance.
The generic SSIS components cannot meet this requirement. You need to code for this e.g. in an SSIS Script task.
I would code that script to read through the file looking for that header row ID Name Address, and then write that line and the rest of the file out to a new file.
Then I would load that new file using the SSIS Flat File Source component.
You might be able to avoid a script task if you'd prefer not to use one. I'll offer a few ideas here as it's not entirely clear which will be best from your example data. To some extent it's down to personal preference anyway, and also the different ideas might help other people in future:
Convert ID and ignore failures: Set the file source so that it expects however many columns you're forced into having by the header rows, and simply pull everything in as string data. In the data flow - immediately after the source component - add a data conversion component or conditional split component. Try to convert the first column (with the ID) into a number. Add a row count component and set the error output of the data conversion or conditional split to be redirected to that row count rather than causing a failure. Send the rest of the data on its way through the rest of your data flow.
This should mean you only get the rows which have a numeric value in the ID column - but if there's any chance you might get real failures (i.e. the file comes in with invalid ID values on rows you otherwise would want to load), then this might be a bad idea. You could drop your failed rows into a table where you can check for anything unexpected going on.
Check for known header values/header value attributes: If your header rows have other identifying features then you could avoid relying on the error output by simply setting up the conditional split to check for various different things: exact string matches if the header rows always start with certain values, strings over a certain length if you know they're always much longer than the ID column can ever be, etc.
Check for configurable header values: You could also put a list of unacceptable ID values into a table, and then do a lookup onto this table, throwing out the rows which match the lookup - then if you need to update the list of header values, you just have to update the table and not your whole SSIS package.
Check for acceptable ID values: You could set up a table like the above, but populate this with numbers - not great if you have no idea how many rows might be coming in or if the IDs are actually unique each time, but if you're only loading in a few rows each time and they always start at 1, you could chuck the numbers 1 - 100 into a table and throw away and rows you load which don't match when doing a lookup onto this table.
Staging table: This is probably the way I'd deal with it if I didn't want to use a script component, but in part that's because I tend to implement initial staging tables like this anyway, and I'm comfortable working in SQL - so your mileage may vary.
Pick up the file in a data flow and drop it into a staging table as-is. Set your staging table data types to all be large strings which you know will hold the file data - you can always add a derived column which truncates things or set the destination to ignore truncation if you think there's a risk of sometimes getting abnormally large values. In a separate data flow which runs after that, use SQL to pick up the rows where ID is numeric, and carry on with the rest of your processing.
This has the added bonus that you can just pick up the columns which you know will have data you care about in (i.e. columns 1 through 3), you can do any conversions you need to do in SQL rather than in SSIS, and you can make sure your columns have sensible names to be used in SSIS.
i am trying to retrieve information from the database but i dont know how number of the rows will be retrieved. information will be retrieved from the following statement.
SELECT * from pgw_alarm WHERE (((pgw_alarm.sev)='0 0 0') AND ((pgw_alarm.month)='"+mon+"'));
now i want to display them to user using table since i dont know how many rows will be there i am unable to create table. is there a way to increase the number of rows in table if so how if not what is the alternative way to display them.
thank you.
In pure HTML this is not possible, however a number of technologies will solve this issue for you. Depending on your server side language, such as JSP or ASP you may be able to dynamically create the rows on the server. Another possibility is to retrieve the rows via Ajax and use javascript to dynamically create dom elements that correspond with each row and append them.
The server-side developer should define a loop in the provided html template. Thus any number of rows can be displayed.
Alternatively the server-side developer should provide a gateway URL which returns JSON formatted data, which can be accessed through AJAX call and then any number of rows generated via JavaScript.