So, I have my current bot (Yes, this bot is entirely legal. It's just a web scraper-esque bot) extracting data from a web page in a loop. But, at the beginning of the loop, I want it to be able to look through the previously-extracted data and compare it to the data on the page to choose what I want to do next. Is this possible?
For example:
Data is saved like this: name1,100,name2,200 Where names can be any string of letters, and the numbers can be any string of numbers.
To give you an idea on what the page looks like: It has two names, each with values under them, and buttons. The names and values change every minute or so.
First, we check the current names on the page. (There are two names on the page)
Next, we compare them to our database to see if they reside in it.
If they both do, we check the numbers after the name
If the first names number is higher than the seconds, we click the first names button
If the second names number is higher than the firsts, we click the second names button
Now we wait twenty seconds
Now, new values have appeared for both names. We find the mean of the new values presented to us and the old values, by adding them together and dividing by two.
What I really need help on is how to search through the CSV file for the names and then numbers after the names, and then how to save data to the same spot as the old data was from.
I know that this is easily possible in most programming languages (Java, C++, etc etc) but I wasn't sure if it would be possible in iMacros.
you can't save data in same csv as you use for search in imacros.
in regular imacros you do not have loop inside loop and IF statements as well, so you need to switch to javascript or similar.
extracting data from csv is simple: http://wiki.imacros.net/!COLn
Related
I have a settings file that contains a single string. Only one setting can be set at one time but you can have multiple saved settings. I need to be able to reference the setting by name, category, and subcategory. For example, the category is character and the subcategory is what 2 spells are selected for that character. I want to track them by unique name but also be able to look up a character and spell setting as well.
My current solution is to have a directory with each category and then spell combinations sub folders with the file being stored in the last folder with the title of the setting as the file name. However, my problem is that it's very slow and bulky.
I thought I could store all the information in one file but then I have to search each file when a character is selected. There may not be an additional setting for a particular combination either, which means it just stays whatever it was originally.
In summary, I want to list a string by name or character and each character has a subcategory that has a two string combination. Does anyone have any ideas on the best method to approach this problem?
I also thought about using one file in a Json strong and just deserializing and querying it, but the file could be extremely large. I am working on the program in vb.net.
I can take this down if it's useless. I used one file as a map to the rest of the larger files. The map has the basic metadata as well to prevent me from having to search all the files. I am not sure if it is the best answer but it did work for me. I can take this question down if no one finds this useful.
I would like to get more or less what is discussed in this question.
What more? I would like to redirect the output to a file.
What less? The SHOW columns FROM your-table; actually returns a lot of stuff. In addition to column names it also shows i.e. type.
Ideally, in the end, I would like a plain text file listing one column name per line.
I have job in Talend that is designed to bring together some data from different databases: one is a MySQL database and the other a MSSQL database.
What I want to do is match a selection of loan numbers from the MySQL database (about 82,000 loan numbers) to the corresponding information we have housed in the MSSQL database.
However, the tables in MSSQL to which I am joining the data from MySQL are much larger (~ 2 million rows), are quite wide, and thus cost much more time to query. Ideally I could perform an inner join between the two tables based on the loan number, but since they are in different databases this is not possible. The inner join that is performed inside a tMap occurs after the Lookup input has already returned its data set, which is quite large (especially since this particular MSSQL query will execute a user-defined function for each loan number).
Is there any way to create a global variable out of the output from the MySQL query (namely, the loan numbers selected by the MySQL query) and use that global variable as an IN clause in the MSSQL query?
This should be possible. I'm not working in MySQL but I have something roughly equivalent here that I think you should be able to adapt to your needs.
I've never actually answered a Stackoverflow question and while I was typing this the page started telling me I need at least 10 reputation to post more than 2 pictures/links here and I think I need 4 pics, so I'm just going to write it out in words here and post the whole thing complete with illustrations on my blog in case you need more info (quite likely, I should think!)
As you can see, I've got some data coming out of the table and getting filtered by tFilterRow_1 to only show the rows I'm interested in.
The next step is to limit it to just the field I want to use in the variable. I've used tMap_3 rather than a tFilterColumns because the field I'm using is a string and I wanted to be able to concatenate single quotes around it but if you're using an integer you might not need to do that. And of course if you have a lot of repetition you might also want to get a tUniqueRows in there as well to save a lot of unnecessary repetition
The next step is the one that does the magic. I've got a list like this:
'A1'
'A2'
'B1'
'B2'
etc, and I want to turn it into 'A1','A2','B1','B2' so I can slot it into my where clause. For this, I've used tAggregateRow_1, selecting "list" as the aggregate function to use.
Next up, we want to take this list and put it into a context variable (I've already created the context variable in the metadata - you know how to do that, right?). Use another tMap component, feeding into a tContextLoad widget. tContextLoad always has two columns in its schema, so map the output of the tAggregateRows to the "value" column and enter the name of the variable in the "key". In this example, my context variable is called MyList
Now your list is loaded as a text string and stored in the context variable ready for retrieval. So open up a new input and embed the variable in the sql code like this
"SELECT distinct MY_COLUMN
from MY_SECOND_TABLE where the_selected_row in ("+
context.MyList+")"
It should be as easy as that, and when I whipped it up it worked first time, but let me know if you have any trouble and I'll see what I can do.
Before I look any further on the internet to try and solve this, is actually possible to do this:
There are two list-boxes with data been displayed. The first list-box shows results from a SQL query. i.e. bus, cab, plane, lorry. The user can select certain data and then add it to listbox2. listbox2 could have the following data: bus, cab, plane in it. what I am trying to do is this:
None of the data has been selected, when an export button is clicked, it loops through listbox2, gets the data from the list-box (Bus, Cab, Plane) and stores it in an array, what I want to do next, is query it in a SQL statement to get the results back to be exported to excel. I can do the SQL part and exporting.
I thought it I would be able to do the code for this, but the amount of different loops and other methods I try it complains. I can get it to count how many rows there are in listbox2, but not pull the data from it.
Looking on the internet the majority of information I seem to find is when it requires data to be selected first - and very little that is based on vba in access, as VBA/VBA Excel is different from the VBA in access so I have found (Urgh)
I'm not expecting someone to give me a coded solution to this, I just want to know before I waste any more time, is this actually possible to achieve?
I have created a CSV from a set of files in a directory that are numbered incrementally:
img1_1.jpg, img1_2.jpg ... img1_1999.jpg, img1_2000.jpg
The CSV output is like so:
filename, datetime
eg:
img1_1.JPG,2011-05-11 09:16:33.000000000
img1_3.jpg,2011-05-11 10:10:55.000000000
img1_4.jpg,2011-05-11 10:17:31.000000000
img1_6.jpg,2011-05-11 10:58:37.000000000
The problem is, there are a number of files missing in the listing, as some of the files don't exist. As a result, when imported, the actual row number does not match the file number.
Can anyone think of a reasonably efficient way to insert the missing rows so that the row number and filename matches up other than manually inserting rows for the missing ones? (There are over 800 missing rows).
Background
A previous programmer developed an uploader script and did not save the creation time of the mysql record in the database. I figured the easiest way to find the creation time for the majority of the records would be to output a directory listing of all the files and combine them in a spreadsheet.
You exactly need to do what you write in your comment to answer #tadman.
A text parser script to inject the missing lines with e.g. a date/time value that reflects the record is an empty one, i.e. there is no real data is behind it (e.g. date it to 1950-01-01 00:00:00). When it is done, bulk import the CSV.I think this must be the best and most efficient solution.
Also, think about any future insert/delete/update events might occur to your data.
That would possibly break the chain you initially have had, so you might prefer instead, to introduce a numeric field for the jpegs IDs (and index that field), and leave the PK "as is" (auto increment).
In this case you can avoid CSV manipulation, as well as being chained to your AUTO PK (means: you will not get in trouble if a new jpeg arrives with an ID which was previously deleted, or existing ID, etc).
So the solution really depends on how you want to use this table in the future. If you give more details, I am sure the community can come up with even more ideas.
If it's a one-time thing, it might be easiest to open up your csv in a spreadsheet.
If your table above is in sheet1, you could put something like the following in sheet2 (this is openoffice, but there are similar functions for Excel)
pre_filename | filename | datetime
img1_1 | = A2&".JPG" | =OFFSET(Sheet1.$B$1;MATCH(B2;Sheet1.$A$2:$A$4;0);0)
You should be able to select the three cells above and drag them down to however many you need.