The format of data is something like this:
TGCCACAGGTTCCACACAACGGGACTTGGTTGAAATATTGAGATCCTTGGGGGTCTGTTAATCGGAGACAGTATCTCAACCGCAATAAACCC
GTTCACGGGCCTCACGCAACGGGGCCTGGCCTAGATATTGAGGCACCCAACAGCTCTTGGCCTGAGAGTGTTGTCTCGATCACGACGCCAGT
TGCCACAGGTTCCACACAACGGGACTTGGTTGAAATATTGAGATCCTTGGGGGTCTGTTAATCGAAGACAGTATCTCAACCGCAATAAACCT
TGCCACAGGTTCCACACAACGGGACTTGGTTGAAATATTGAGATCCTTGGGGGTCTGTTAATCGAAGACAGTATCTCAACCGCAATAAACCT
Each line contains one sequence, I want to make a pair of (key ,value), key is one sequence and value is 1. Then use reduce_by_key to count the number of each sequence.
But I found that thrust::host_vector can only store one sequence, if I push_back the 2nd sequence the program crashed.
Here is my code:
int main()
{
ifstream input_subset("subset.txt");
thrust::host_vector < string > h_output_subset;
string s;
while (getline(input_subset, s)) {
h_output_subset.push_back(s);
}
cout << h_output_subset.size() << endl;
return 0;
}
Is that possible to store all of data in a host_vector or a device_vector? Or is there any way to solve this problem?
The host_vector segfault was confirmed as a bug in thrust::uninitialised_copy and a patch has been applied to fix it.
The problem doing this with a device_vector is a genuine limitation of CUDA (no std::string support) and can't be avoided. An alternative would be to use a fixed length char[] array as a data member in a device_vector, or use a single large device_vector to hold all the string data, with a second device_vector holding the starting index of each sub-string within the character array.
Related
I want to try out Critcl to enhance memory performance using a Z-order curve for a 2d-grid. What I need from Critcl is allocation, setter, getter and some size info. Reading about the Critcl ByteArray and examples does not make me confident on how to do it.
How do I create and return a ByteArray (i.e. Z-order curve)?
Any caveats I should know about when using ByteArray?
According to the documentation, you should be using the bytes type instead (when you get a pointer to a structure that has a len field with the number of bytes in it, and an s field that is the pointer to the actual read only block of bytes. (As a char * and not an unsigned char * for reasons I don't know. And why it isn't const is another mystery to me; there are cases where that's indeed true, but you need to look at the o field to figure that out.)
To return a byte array, you use the object (or object0) result type, and make the object with, for example, Tcl_NewByteArrayObj(), or Tcl_NewObj() and Tcl_SetByteArrayLength().
Here's an example (just the command definition) that does trivial byte reversing (since I don't understand Z-order curves at all):
critcl::cproc example {bytes dataPtr} object0 {
Tcl_Obj *result = Tcl_NewObj();
unsigned char *targetBytes = Tcl_SetByteArrayLength(result, dataPtr->len);
for (int i = 0, j = dataPtr->len - 1; j >= 0; i++, j--) {
targetBytes[i] = (unsigned byte) dataPtr->s[j];
}
return result;
}
Naturally, you'll want to read the Critcl usage guide when getting this to work, and if you're going to produce errors (by returning NULL), remember to set an error message in the interpreter. You can get access to that by using Tcl_Interp* interp as your first pseudo-argument to the command you create with critcl::cproc (it's documented, but easily missed).
I have a somewhat rather simple cuda question that seems like it should be a straight forward operation: removing elements from 1 array based on the value of a 2nd bool array. The steps I take are:
Create a device_vector of bools with the same size as the processed input array.
Call kernel which will set some of the elements from (1) to true
Call remove_if on input array with predicate using processed array from (2).
For each value in the bool array that is set to true, remove the corresponding element from the input array.
What I am seeing is that the input array isn't changed and I am not sure why ?
struct EntryWasDeleted
{
__device__ __host__
bool operator()(const bool ifDeleted)
{ return true; }
};
//This array has about 200-300 elements
//thrust::device_vector<SomeStruct> & arrayToDelete
thrust::device_vector<bool>* deletedEntries =
new thrust::device_vector<bool>(arrayToDelete.size(), false);
cuDeleteTestEntries<<<grid, block>>>( thrust::raw_pointer_cast(arrayToDelete.data()), countToDelete, heapAccess, thrust::raw_pointer_cast(deletedEntries->data()));
cudaDeviceSynchronize();
thrust::remove_if(arrayToDelete.begin(), arrayToDelete.end(), deletedEntries->begin(), EntryWasDeleted());
//I am expecting testEntries to have 0 elements
thrust::host_vector<SomeStruct> testEntries = arrayToDelete;
for( int i = 0; i<testEntries.size(); i++)
{ printf( "%d", testEntries[i].someValue); }
In this sample, I am always returning true in the predicate for testing. However, when I do: testEntries = deletedEntries and output the members. I can validate that deletedEntries is properly filled in with trues and falses.
My expectation would be that testEntries would have 0 elements. But it doesn't and I get an output as if remove_if didn't do anything. ie: the output is showing ALL elements from the input array. I am not sure why? Is there a specific way to remove elements from a device_vector?
So you need to capture the iterator that is being returned from remove_if
thrust::device_vector<SomeStruct>::iterator endIterator =
thrust::remove_if(arrayToDelete.begin(), arrayToDelete.end(),
deletedEntries->begin(), EntryWasDeleted());
Then when you copy data back to the host instead of using thrusts default assignment operator between host and device do this:
thrust::host_vector<SomeStruct> testEntries(arrayToDelete.begin(),endIterator);
As a side note working with arrays of primitives can often be much more efficient. Like can you store the index of your structs in an array instead and operate on those indexes?
I'm afraid to ask questions in case it turns out to be stupid... But I tried to search and don't see the same situation.
I'm retrofitting a std::vector into some existing legacy code that is mostly C style. Our next major release which isn't due for a year or two will jettison a lot of the legacy code. But for now, the way we work is, every project gets recompiled for the customer, depending on specs. Some projects are on Visual Studio 2008, some 2010, etc. My added std::vector code I'm working on has no visible problems when compiled with 2013, but, I get crashes within the STL code when running VS 2008 SP1.
The existing code has a struct, and a fixed size array in it:
#define MAX_REMOTE_CONN 75
typedef struct {
int rno;
int adrs;
bool integ_pending;
} RTUref;
typedef struct {
char device[64];
int port;
RTUref RTU[MAX_REMOTE_CONN];
// more stuff...
} Connect_Info;
So, my basic goal is to get rid of the hard coded size limit to the RTU array. So, I have revised it like this:
class{
public:
int rno;
int adrs;
bool integ_pending;
} RTUref;
typedef std::vector <RTUref> RTUlist;
typedef struct {
char device[64];
int port;
RTUlist RTU;
// more stuff...
} Connect_Info;
The Connect_Info structs are allocated using our own memory manager. Don't know much about it other than it is supposed to be more efficient than use malloc() and free(). I'm guessing that the constructor for RTU doesn't get called since the struct it is contained in data allocated by our own memory manager?
Nevertheless, the code where I size the array, put values into the array all at least seem to work okay. But, when I call .clear() I get a crash from within the STL. And as I said, only if I use 2008. If I use 2013, I don't get that crash.
Assuming pct is a pointer to an allocated Connect_Info structure, the the line:
pct->RTU.clear();
Generates a crash on VS 2008. I am able to resize and add elements to the array. And I even tried to add a check that I don't clear unless the size is greater than zero like so:
if (pct->RTU.size() > 0)
pct->RTU.clear();
And I still get the crash on the clear.
So, I made the educated guess that I need to call a constructor. But, I wasn't quite sure of how to do it. But, in the code where the Connect_Info struct is allocated, I tried to add contructor code like this:
pct->RTU = RTUlist();
It compiles. But, I then get a crash in the STL on that line.
I haven't yet tried to build a small contained test program, as I'm not even sure that I will be able to reproduce the problem without our memory manager. But, I will try if that is what I need to do. I thought maybe someone might see obviously what I'm doing wrong. I'm fairly novice to the STL.
A little background: there is a term in C++ called "POD Type" (or "Plain Old Data Type").
There are verbose rules, but basically things that may do special things on allocations, deallocations, or copies are not POD types. The original Connect_Info was a POD type since it didn't do special things at those times and didn't have any non-POD members.
However, since you added a std::vector (which is not a POD type because it has to do special things at allocation, deallocation, copy, etc (because it allocates memory)), Connect_Info is not a POD type.
POD types can be allocated safely with malloc and deallocated with free since they don't do special things. However, non-POD types cannot (except in exceedingly rare cases which you'll first see after several years of programming C++) be allocated like that.
C only has POD types, so malloc is perfectly acceptable. There are a few options you can do:
int main ( ... )
{
Connect_Info * info = new Connect_Info() ;
std::cout << info->port << std::endl ;
delete info ;
}
Or
Connect_Info * makeOne ()
{
void * ptr = malloc ( sizeof(Connect_Info) ) ;
if ( ! ptr ) return 0 ;
return new (ptr) Connect_Info () ; // "In-Place constructor"
}
void deleteOne ( Connect_Info * info )
{
if ( ! ptr ) return ;
info = info->~Connect_Info() ; // manually call its destructor with the weirdest syntax ever
// Note: I'm not 100% sure this call to 'free' is right because the in-place new can return a different pointer, but I don't know how to the get the original back
free ( static_cast<void*>(info) ) ;
}
int main ( ... )
{
Connect_Info * info = makeOne ()
std::cout << info->port << std::endl ;
deleteOne ( info ) ;
}
If you have boost available (or C++11, which you probably don't), this is a MUCH better option (and only uses header components of boost):
boost::shared_ptr<Connect_Info> makeOne ()
{
return boost::make_shared<Connect_Info> () ;
}
int main ( ... )
{
boost::shared_ptr<Connect_Info> info = makeOne ()
std::cout << info->port << std::endl ;
// nothing else: shared_ptr takes care of that for you
}
(If you have C++11, use std::shared_ptr and std::make_shared)
I am creating a program that can make mysql transactions through C and html.
I have this query string
query = -id=103&-id=101&-id=102&-act=Delete
Extracting "Delete" by sscanf isn't that hard, but I need help extracting the integers and putting them in an array of int id[]. The number of -id entries can vary depending on how many checkboxes were checked in the html form.
I've been searching for hours but haven't found any applicable solution; or I just did not understand them. Any ideas?
Thanks
You can use strstr and atoi to extract the numbers in a loop, like this:
char *query = "-id=103&-id=101&-id=102&-act=Delete";
char *ptr = strstr(query, "-id=");
if (ptr) {
ptr += 4;
int n = atoi(ptr);
printf("%d\n", n);
for (;;) {
ptr = strstr(ptr, "&-id=");
if (!ptr) break;
ptr += 5;
int n = atoi(ptr);
printf("%d\n", n);
}
}
Demo on ideone.
You want to use strtok or a better solution, to tokenize this string with & and = as tokens.
Take a look at cplusplus.com for more information and an example.
This is the output you would get from strtok
Output:
Splitting string "- This, a sample string." into tokens:
This
a
sample
string
Once you figure out how to split them, the next hurdle is to convert the numbers from strings to ints. For this you need to look at atoi or its safer more robust cousin strtol
Most likely I would write a small lexical scanner to tackle the task. Meaning, I would analyze the string one character at a time, according to a regular expression representing the set of possible inputs.
I am using the MySQL C API to query the database and I have the results stored in MYSQL_ROW types. I am able to print the results to the console with
printf("%s", row[0]);
however, according to the MySQL C API documentation, I cannot use them as null-terminated strings.
At the bottom of the function overview, they say I can "extract" the information with mysql_store_result() or mysql_use_result(). However, I am still confused as to how this is done.
Ideally, I want to use the results as a string so I can do stuff like strcmp, but otherwise I definitely need to use the information somehow with those two functions.
Can somebody show me an example of how to do this?
Basically, you call mysql_store_result() or mysql_use_result() to access the result set, the former loads all the rows into memory on the client side, the latter accesses rows one at a time from the server. If you use mysql_use_result(), you need to call mysql_fetch_row() to access each row until the function returns NULL. Each successful call to mysql_fetch_row() will return a MYSQL_ROW which you can use to access the individual field values.
Since the fields are not nul-terminated, you need to use mysql_fetch_lengths() to get the lengths of each of the fields so that you can copy them somewhere else via memcpy, etc.
Since the field values are not nul-terminated you will need to add your own NUL character when you make the copy if you want to use it as a string. Be aware that the field values may contain binary data, so if you do treat it as a string, functions that expect a C string will stop processing data if it encounters a nul-character in the data.
Here is an example from the documentation that should help you put all this together:
MYSQL_ROW row;
unsigned int num_fields;
unsigned int i;
num_fields = mysql_num_fields(result);
while ((row = mysql_fetch_row(result)))
{
unsigned long *lengths;
lengths = mysql_fetch_lengths(result);
for(i = 0; i < num_fields; i++)
{
printf("[%.*s] ", (int) lengths[i],
row[i] ? row[i] : "NULL");
}
printf("\n");
}