SnappyData documentation give an example on how to submit a jar to a cluster:
https://snappydatainc.github.io/snappydata/howto/run_spark_job_inside_cluster/
But what if I need to submit the jar with the same class CreatePartitionedRowTable
multiple times, but with different paramter, say different suffix to append to the names of the tables created, How do I do that?
UPDATE:
To be more precise, say I want to submit the jar with different parameters when I submit the jar, something like this
bin/snappy-job.sh submit
--app-name CreatePartitionedRowTable
--class org.apache.spark.examples.snappydata.CreatePartitionedRowTable
--app-jar examples/jars/quickstart.jar
--lead localhost:8090
--CustomeParam suffix
the additional
--CustomeParam suffix
will be passed in to the job, and the code can pick up this parameter suffix, and appending the suffix to the table names to be created, so that I don't have to modify my code every time that I want to submit the jar with a different suffix.
Update 2:
I just went through the examples and found an example usage:
https://github.com/SnappyDataInc/snappydata/blob/master/examples/src/main/scala/org/apache/spark/examples/snappydata/CreateColumnTable.scala
so basically run like this:
* bin/snappy-job.sh submit
* --app-name CreateColumnTable
* --class org.apache.spark.examples.snappydata.CreateColumnTable
* --app-jar examples/jars/quickstart.jar
* --lead [leadHost:port]
* --conf data_resource_folder=../../quickstart/src/main/resources
and use config to get the customized parameter.
Each time you submit your app jar with snappy-job.sh it will create a new Job and run it. It could be the same jar with different content. Do you see any exception or the modified class (CreatePartitionedRowTable) is not getting picked ?
Related
Disclaimer: new to SSIS and Active Directory
I have a need to extract all users within a particular Active Directory (AD) domain and import them into Excel. I have followed this: https://www.itnota.com/query-ldap-in-visual-studio-ssis/ in order to create my SSIS package. My SQL is:
LDAP://DC=JOHN,DC=JANE,DC=DOE;(&(objectCategory=person)(objectClass=user)(name=a*));Name,sAMAccountName
As you know there is a 1,000 row limit when pulling from the AD. In my SQL I currently have (name=a*) to test the process and it works. I need to know how to setup a loop with variables to pull all records and import into Excel (or whatever you experts recommend). Also, how do I know what the other field names are that are available to pull?
Thanks in advance.
How do I see what's in Active Directory
Tool recommendations are off topic for the site but a tool that you can download, no install required, is AD Explorer It's a MS tool that allows you to view your domain. Highly recommend people that need to see what's in AD use something like this as it shows you your basic structure.
What's my domain controller?
Start -> Command Prompt
Type set | find /i "userdnsdomain" and look for USERDNSDOMAIN and put that value in the connect dialog and I save it because I don't want to enter this every time.
Search/Find and then look yourself up. Here I'm going to find my account by using my sAMAccountName
The search results show only one user but there could have been multiples since I did a contains relationship.
Double clicking the value in the bottom results section causes the under pane window to update with the details of the search result.
This is nice because while the right side shows all the properties associated to my account, it's also updated the left pane to navigate to the CN. In my case it's CN=Users but again, it could be something else in your specific environment.
You might discover an interesting categorization for your particular domain. At a very large client, I discovered that my target users were all under a CN
(Canonical Name, I think) so I could use that in my AD query.
There are things you'll see here that you sure would like to bring into a data flow but you won't be able to. Like the memberOf that's a complex type and there's no equivalent in the data flow data types for it. I think Integer8 is also something that didn't work.
Loop the loop
The "trick" here is that we'll need to take advantage of the
The name of the AD provider has changed since I last looked at this. In VS 2017, I see the OLE DB Provider name as "OLE DB Provider for Microsoft Directory Service"
Put in your query and you should get results back. Let that happen so the metadata is set.
An ADO.NET source does not support parameterization as the OLE DB does. However, you can apply an Expression on the Data Flow which surfaces the component and that's what we'll do.
Click out of the Data Flow and back into the Control Flow and right click on the Data Flow and select Properties. In that properties window, find Expressions and click the ellipses ... Up pops the Property Expressions Editor
Find the ADO.NET source under Property and in the Expressions section, click the Ellipses.
Here, we'll use your same source query just to prove we're doing the right things
"LDAP://DC=JOHN,DC=JANE,DC=DOE;(&(objectCategory=person)(objectClass=user)(name=" + "a" + "*));Name,sAMAccountName"
We're doing string building here so the problem we're left to solve is how we can substitute something for the "a" in the above query.
The laziest route would be to
Create an SSIS variable of type String called CurrentLetter and initialize it to a
Update the expression we just created to be "LDAP://DC=JOHN,DC=JANE,DC=DOE;(&(objectCategory=person)(objectClass=user)(name=" + #[USer::CurrentLetter] + "*));Name,sAMAccountName"
Add a Foreach Loop Container (FELC) to your Control Flow.
Configure the FELC with an enumerator of "Foreach Item Enumerator"
Click the Columns...
Click Add (this results in Column 0 with data type String) so click OK
Fill the collection with each letter of the alphabet
In the Variable Mappings tab, assign Variable User::CurrentLetter to Index 0
Click OK
Old blog posts on the matter because I like clicks
https://billfellows.blogspot.com/2011/04/active-directory-ssis-data-source.html
http://billfellows.blogspot.com/2013/11/biml-active-directory-ssis-data-source.html
I'm getting this exception when trying to ->write() a DataObject called 'ModelSheet', it says the name should be Models\ModelSheet instead of ModelSheet only (i am under the same namespace (Models) and even try with an use statement)
Hi Guilherme and welcome to stackoverflow,
it seems that the ClassName saved to your database record does not match your PHP classname.
When changing classnames (adding or changing a namespace is changing the classname), you need to update the database to reflect this changes, as the classname is saved in the DB, so Silverstripe knows which PHP-Object is related to the data record.
If you used Silverstripe's upgrader tool, you should have an .upgrade.yml in your module's directory (e.g. in app or mysite). If not, you can add it manually (see e.g. https://github.com/wernerkrauss/silverstripe-onepage/blob/master/.upgrade.yml as a random example). The structure is like
mappings:
OldClassName: My\Namespace\NewClassname
After that all you need is to run dev/build/?flush and your database should be updated.
Here is my test plan structure.
User Login
Runtime Controller
while controller !<> EOF
CSV dataset (items to add)
search and add to cart
Click cart.
Proceed to check out
Order submit.
Beanshell sampler to close CSV
User Logout.
I want each thread to read the csv till EOF and add these items to cart, hence I used the sharing
mode as current thread.Since add to cart and order submission is getting repeated for the test
duration I am closing the file and resetting the variable after order submit so that next iteration
will again start to read from beginning.
The beanshell code is :
import org.apache.jmeter.## Heading ##services.FileServer;
FileServer.getFileServer().closeFile("Scripts_Helan\\DSOrderParts.csv");
String pPartNum = vars.get("pPartNum");
vars.put("pPartNum", "");
But when I run the test Jmeter log is showing file name as
Stored: Scripts_Helan\DSOrderParts.csv Alias: Scripts_Helan\DSOrderParts.csv#1309262272
Don't I have to use the Alias in closeFile? How can I get it?[enter image description here][1]
[enter image description here][1]
I don't exactly get why you are using beanshell code here.
You can handle the "start all over when done" part with setting up the Thread Group accordingly
You can handle that "stop thread at end of file" part with setting up the CSV Data Set Config accordingly
Please clarify what makes handling the file in beanshell code neccessary.
I've multiple http samples in Test Plan but i want to run only one request for multiple times based on csv data set config and remaining http requests will run once particular sample is complete all the iteration based on csv data set config.
In above image i've 5 http samples and i've csv data set for first request,once first request is run for entire csv data set test data then another samples should run,please provide solution how to perform this type of operation in jmeter.
you can use Loop Control and provide the number of hits you needed.
To go to Loop Control:
Right click on the Thread/ThreadGroup
Click on Add
select the Logic controller
Select the Loop Controller
According to the problem, there is a need to execute a particular request to execute in a loop. The loop could be organized in the following way.
-> Thread Group
-> Loop Controller
-->Particular Request
-> Other Request
The CSV Dataset could be found in Config Element and should contain a path to a particular file with data (e.g. accounts.csv).
Both elements could be located in
Loop controller: Add ->Logic Controller -> Loop Controller
CSV Data Set : Add -> Config Elements -> CSV Data Set Config.
The loop can run for the exact amount of records in a file if
Loop Count is equal to a number of strings in the accounts.csv
Forever, it could be stopped by Test Duration automatically or by Stop button.
The CSV Data Set config should have a path to a file with data.
If a file contains headers, then Variables will be assigned based on column names from first string, otherwise, it is necessary to provide variables names in configuration.
ANSWER: If you use 9 variables and all of them are set as header names in a file, then don't provide them in CSV Data Set Config. They will be created automatically, so you can use all 9 variables through names in your file.
If headers are not provided, set 9 all in a Variable Names field, using comma as separator.
Configure your CSV Data Set Config like:
Reference name: foo
Recycle on EOF: false
Stop thread on EOF: false
Put the CSV Data Set Config under the While Controller and put ${__javaScript("${foo}" != "<EOF>",)} into "Condition" input
Add If Controller under the While Controller and put "${foo}" != "<EOF>" into the "Condition" input
Put your first request as a child of the If Controller
Replace foo with your actual JMeter Variable reference name everywhere in the script.
This way JMeter will loop until there are entries in the CSV file and will proceed with the remaining requests when the end of CSV file will be reached.
See Reading all Values from the CSV and Continue chapter of the Using the While Controller in JMeter article for more detailed information on configuring JMeter for your scenario.
You can verify with only one variable and Rest variable you don't required in If and while controller.
The config api within Trac is a pretty straight forward way of storing and loading settings from the config file. However, I haven't yet found a method to delete a line, or to delete all lines from a category that have no value assigned. So far I can only set the values to False or an empty string to deactivate them.
What I want to do is remove all values from my plugin's category that were set manually in the file and are not accepted by my script, or were set to False or no value by the script. In both ways, the whole line should be removed from the config so that it doesn't contain a lot of lines that look like this:
some_setting =
Edit
I just found out that I can assign None as value, which comments the setting, and the next time Trac writes to the config file it removes the commented line. This is a possible workaround, but I'm still interested in whether a direct method exists.
You can use Configuration.remove(...). Within a Component class call self.config.remove('section', 'key').