How to execute a scenario using data from the previous scenario? - stress-testing

I'd like to execute two scenarios that should be executed one after another and the data "produced" by the first scenario should be used as base for the second scenario.
So a case could be for example clearing of a credit card. The first scenarios is there to authorize/reserve of a certain amount on the card:
val auths = scenario("auths").during(durationInMinutes minutes) {
feed(credentials)
.feed(firstNames)
.feed(lastNames)
.feed(cards)
.feed(amounts)
.exec(http("send auth requests")
.post(...)
.check(...))}
The second one is there to capture/take the amount from the credit card:
val caps = scenario("caps").during(durationInMinutes minutes) {
feed(credentials)
.feed(RESPONSE_IDS_FROM_PREVIOUS_SCENARIO)
.exec(http("send auth requests")
.post(...)
.check(...))}
I initially thought about using the saveAs(...) option on check but I figured out that the saved field is only valid for the given session.
So basically I want to preserve the IDs I got from the auths scenario and use them in the caps scenario.
I cannot execute both steps in one scenario though (saveAs would work for that) because I have different requirement for both scenarios.

Quoting the documentation: "Presently our Simulation is one big monolithic scenario. So first let us split it into composable business processes, akin to the PageObject pattern with Selenium. This way, you’ll be able to easily reuse some parts and build complex behaviors without sacrificing maintenance." at gatling.io/Advanced Tutorial
Thus your there is no build-in mechanism for communication between scenarios (AFAIK). Recommendation is to structure your code that way that you can combine your calls to URIs subsequently. In your case (apart from implementation details) you should have something like this:
val auths = feed(credentials)
.feed(firstNames)
.feed(lastNames)
.feed(cards)
.feed(amounts)
.exec(http("send auth requests")
.post(...)
.check(...) // extract and store RESPONSE_ID to session
)
val caps = exec(http("send auth requests")
.post(...) // use of RESPONSE_ID from session
.check(...))
Then your scenario can look something like this:
val scn = scenario("auth with caps").exec(auths, caps) // rest omitted
Maybe even better way to structure your code is to use objects. See mentioned tutorial link.
More illustrative example (which compiles, but I didn't run it while domain is foo.com):
import io.gatling.core.Predef._
import io.gatling.http.Predef._
class ExampleSimulation extends Simulation {
import scala.util.Random
import scala.concurrent.duration._
val httpConf = http.baseURL(s"http://foo.com")
val emails = Iterator.continually(Map("email" -> (Random.alphanumeric.take(20).mkString + "#foo.com")))
val names = Iterator.continually(Map("name" -> Random.alphanumeric.take(20).mkString))
val getIdByEmail = feed(emails)
.exec(
http("Get By Email")
.get("/email/$email")
.check(
jsonPath("userId").saveAs("anId")
)
)
val getIdByName = feed(names)
.exec(
http("Get By Name")
.get("/name/$name")
.check(
jsonPath("userId").is(session =>
session("anId").as[String]
)
)
)
val scn = scenario("Get and check user id").exec(getIdByEmail, getIdByName).inject(constantUsersPerSec(5) during (5.minutes))
setUp(scn).protocols(httpConf)
}
Hope it is what you're looking for.

Related

Inconsistent behaviour when attempting to write Dataframe to CSV in Apache Spark

I'm trying to output the optimal hyperparameters for a decision tree classifier I trained using Spark's MLlib to a csv file using Dataframes and spark-csv. Here's a snippet of my code:
// Split the data into training and test sets (10% held out for testing)
val Array(trainingData, testData) = assembledData.randomSplit(Array(0.9, 0.1))
// Define cross validation with a hyperparameter grid
val crossval = new CrossValidator()
.setEstimator(classifier)
.setEstimatorParamMaps(paramGrid)
.setEvaluator(new BinaryClassificationEvaluator)
.setNumFolds(10)
// Train model
val model = crossval.fit(trainingData)
// Find best hyperparameter combination and create an RDD
val bestModel = model.bestModel
val hyperparamList = new ListBuffer[(String, String)]()
bestModel.extractParamMap().toSeq.foreach(pair => {
val hyperparam: Tuple2[String,String] = (pair.param.name,pair.value.toString)
hyperparamList += hyperparam
})
val hyperparameters = sqlContext.sparkContext.parallelize(hyperparamList.toSeq)
// Print the best hyperparameters
println(bestModel.extractParamMap().toSeq.foreach(pair => {
println(s"${pair.param.parent} ${pair.param.name}")
println(pair.value)
}))
// Define csv path to output results
var csvPath: String = "/root/results/decision-tree"
val hyperparametersPath: String = csvPath+"/hyperparameters"
val hyperparametersFile: File = new File(hyperparametersPath)
val results = (hyperparameters, hyperparametersPath, hyperparametersFile)
// Convert RDD to Dataframe and write it as csv
val dfToSave = spark.createDataFrame(results._1.map(x => Row(x._1, x._2)))
dfToSave.write.format("csv").mode("overwrite").save(results._2)
// Stop spark session
spark.stop()
After finishing a Spark job, I can see the part-00*... and _SUCCESS files inside the path as expected. However, though there are 13 hyperparameters total in this case (confirmed by printing them on screen), cat-ing the csv files shows not every hyperparameter was written to csv:
user#master:~$ cat /root/results/decision-tree/hyperparameters/part*.csv
checkpointInterval,10
featuresCol,features
maxDepth,5
minInstancesPerNode,1
Also, the hyperparameters that do get written change in every execution. This is executed on a HDFS-based Spark cluster with 1 master and 3 workers that have exactly the same hardware. Could it be a race condition? If so, how can I solve it?
Thanks in advance.
I think I figured it out. I expected dfTosave.write.format("csv")save(path) to write everything to the master node, but since the tasks are distributed to all workers, each worker saves its part of the hyperparameters to a local CSV in its filesystem. Because in my case the master node is also a worker, I can see its part of the hyperparameters. The "inconsistent behaviour" (i.e. seeing different parts in each execution) is caused by whatever algorithm Spark uses for distributing partitions among workers.
My solution will be to collect the CSVs from all workers using something like scp or rsync to build the full results.

Fetch the response from sql, store it in a object and use conditions?

I have two sql statements to be executed with a validity check. My need is that I execute the 1st query and store the response in one object and check the object is empty or not and execute the second query if it is not empty.
So, I have tried something like
In rolerepository.scala=>
override val allQuery = s"""
select UserRoles.* from
(select CASE rbac.roleTypeID
ELSE rbac.name JOIN dirNetworkInfo ni
ON UserRoles.PersonID = ni.PersonID
where ni.Loginname = {loginName}
and UserRoles.roleName in ( 'Business User ','Administrator')"""
(This is just some sample of the query - it is not fully written here.)
Then I map it to an object with model class written outside
override def map2Object(implicit map: Map[String, Any]):
HierarchyEntryBillingRoleCheck = {
HierarchyEntryBillingRoleCheck(str("roleName"), oint("PersonID")) }
Then I have written the getall method to execute the query
override def getAll(implicit loginName: String):
Future[Seq[HierarchyEntryBillingRoleCheck]] = {
doQueryIgnoreRowErrors(allQuery, "loginName" -> loginName) }
Then I have written the method to check whether the response from the 1st sql is empty or not. This is were I'm stuck and not able to proceed further.
def method1()= {
val getallresponse = HierarchyEntryBillingRoleCheck
getallresponse.toString
if (getallresponse != " ")
billingMonthCheckRepository.getrepo()
}
I am getting an error (type mismatch) in last closing brace and I don't know what other logic can be used here.
Can any one of you please explain and give me some solution for this?
And i also tried to use for loop in controller but not getting how to do that.
i tried ->
def getAll(implicit queryParams: QueryParams,
billingMonthmodel:Seq[HierarchyEntryBillingRoleCheck]):
Action[AnyContent] = securityService.authenticate() { implicit request
=> withErrorRecovery { req =>
toJson {
repository.getAll(request.user.loginName)
for {
rolenamecheck <- billingMonthmodel
}yield rolenamecheck
}}}}
You don't say which db access method you are using. (I'm assuming anorm). One way of approaching this is:
Create a case class matching your table
Create a parser matching your case class
use Option (or Either) to return a row for a specific set of parameters
For example, perhaps you have:
case class UserRole (id:Int, loginName:String, roleName:String)
And then
object UserRole {
val sqlFields = "ur.id, ur.loginName, ur.roleName"
val userRoleParser = {
get[Int]("id") ~
get[String]("loginName") ~
get[String]("roleName") map {
case id ~ loginName ~ roleName => {
UserRole(id, loginName, roleName)
}
}
}
...
The parser maps the row to your case class. The next step is creating either single row methods like findById or findByLoginName and multi-row methods, perhaps allForRoleName or other generic filter methods. In your case there might (assuming a single role per loginName) be something like:
def findByLoginName(loginName:String):Option[UserRole) = DB.withConnection { implicit c =>
SQL(s"select $sqlFields from userRoles ur ...")
.on('loginName -> loginName)
.as(userRoleParser.singleOpt)
}
The .as(parser... is key. Typically, you'll need at least:
as(parser.singleOpt) which returns an Option of your case class
as(parser *) which returns a List of your case class (you'll need this if multiple roles could exist for a login
as(scalar[Long].singleOpt) which returns an Option[Long] and which is handy for returning counts or exists values
Then, to eventually return to your question a little more directly, you can call your find method, and if it returns something, continue with the second method call, perhaps like this:
val userRole = findByLoginName(loginName)
if (userRole.isDefined)
billingMonthCheckRepository.getrepo()
or, a little more idiomatically
findByLoginName(loginName).map { userRole =>
billingMonthCheckRepository.getrepo()
...
I've shown the find method returning an Option, but in reality we find it more useful to return an Either[String,(your case class)], and then the string contains the reason for failure. Either is cool.
On my version of play (2.3.x), the imports for the above are:
import play.api.db._
import play.api.Play.current
import anorm._
import anorm.SqlParser._
You're going to be doing this sort of thing a lot so worth finding a set of patterns that works for you.
WOW I don't know what's happening with the formatting here, I am really attempting to use the code formatter on the toolbar but I don't know why it won't format it, even when pressed multiple times. I invite the community to edit my code formatting because I can't figure it out. Apologies to OP.
Because I find Play's documentation to be very tough to trudge through if you're unfamiliar with it, I won't just leave a link to it only.
You have to inject an instance of your database into your controller. This will then give it to you as a global variable:
#Singleton
class LoginRegController #Inject()(**myDB: Database**, cc: ControllerComponents ) {
// do stuff
}
But, it's bad practice to actually use this connection within the controller, because the JDBC is a blocking operation, so you need to create a Model which takes the db as a parameter to a method. Don't set the constructor of the object to take the DB and store it as a field. For some reason this creates connection leaks and the connections won't release when they are done with your query. Not sure why, but that's how it is.
Create a Model object that you will use to execute your query. Instead of passing the DB through the object's constructor, pass it through the method you will create:
object DBChecker {
def attemptLogin(db:Database, password:String): String = {
}}
In your method, use the method .withConnection { conn => to access your JDBC connection. So, something like this:
object DBChecker {
def attemptLogin(db:Database, password:String):String = {
var username: String = ""
db.withConnection{ conn =>
val query:String = s"SELECT uploaded_by, date_added FROM tableName where PASSWORD = $password ;"
val stmt = conn.createStatement()
val qryResult:ResultSet = stmt.executeQuery(query)
// then iterate over your ResultSet to get the results from the query
if (qryResult.next()) {
userName = qryResult.getString("uploaded_by")
}
}
}
return username
}
// but note, please look into the use of PreparedStatement objects, doing it this way leaves you vulnerable to SQL injection.
In your Controller, as long as you import the object, you can then call that object's methods from your controller you made in Step 1.
import com.path.to.object.DBChecker
#Singleton
class LoginRegController #Inject()(myDB: Database, cc: ControllerComponents ) { def attemptLogin(pass:String) = Action {
implicit request: Request[AnyContent] => {
val result:String = DbChecker.attemptLogin(pass)
// do your work with the results here
}

How to call Stored Procedures and defined functions in MySQL with Slick 3.0

I have defined in my db something like this
CREATE FUNCTION fun_totalInvestorsFor(issuer varchar(30)) RETURNS INT
NOT DETERMINISTIC
BEGIN
RETURN (SELECT COUNT(DISTINCT LOYAL3_SHARED_HOLDER_ID)
FROM stocks_x_hldr
WHERE STOCK_TICKER_SIMBOL = issuer AND
QUANT_PURCHASES > QUANT_SALES);
END;
Now I have received an answer from Stefan Zeiger (Slick lead) redirecting me here: User defined functions in Slick
I have tried (having the following object in scope):
lazy val db = Database.forURL("jdbc:mysql://localhost:3306/mydb",
driver = "com.mysql.jdbc.Driver", user = "dev", password = "root")
val totalInvestorsFor = SimpleFunction.unary[String, Int]("fun_totalInvestorsFor")
totalInvestorsFor("APPLE") should be (23)
Result: Rep(slick.lifted.SimpleFunction$$anon$2#13fd2ccd fun_totalInvestorsFor, false) was not equal to 23
I have also tried while having an application.conf in src/main/resources like this:
tsql = {
driver = "slick.driver.MySQLDriver$"
db {
connectionPool = disabled
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://localhost/mydb"
}
}
Then in my code with #StaticDatabaseConfig("file:src/main/resources/application.conf#tsql")
tsql"select fun_totalInvestorsFor('APPLE')" should be (23)
Result: Error:(24, 9) Cannot load #StaticDatabaseConfig("file:src/main/resources/application.conf#tsql"): No configuration setting found for key 'tsql'
tsql"select fun_totalInvestorsFor('APPLE')" should be (23)
^
I am also planning to call stored procedures that return one tuple of three values, via sql"call myProc(v1).as[(Int, Int, Int)]
Any ideas?
EDIT: When making
sql""""SELECT COUNT(DISTINCT LOYAL3_SHARED_HOLDER_ID)
FROM stocks_x_hldr
WHERE STOCK_TICKER_SIMBOL = issuer AND
QUANT_PURCHASES > QUANT_SALES""".as[(Int)]
results in SqlStreamingAction[Vector[Int], Int, Effect] instead of the suggested DBIO[Int] (from what I infer) suggested by the documentation
I've been running into exactly the same problem for the past week. After some extensive research (see my post here, I'll be adding a complete description of what I've done as a solution), I decided it can't be done in Slick... not strictly speaking.
But, I'm resistant to adding pure JDBC or Anorm into our solution stack, so I did find an "acceptable" fix, IMHO.
The solution is to get the session object from Slick, and then use common JDBC to manage the stored function / stored procedure calls. At that point you can use any third party library that makes it easier... although in my case I wrote my own function to set up the call and return a result set.
val db = Database.forDataSource(DB.getDataSource)
var response: Option[GPInviteResponse] = None
db.withSession {
implicit session => {
// Set up your call here... (See my other post for a more detailed
// answer with an example:
// procedure is eg., "{?=call myfunction(?,?,?,?)}"
val cs = session.conn.prepareCall(procedure.toString)
// Set up your in and out parameters here
// eg. cs.setLong(index, value)
val result = cs.execute()
val rc = result.head.asInstanceOf[Int]
response = rc match {
// Package up the response to the caller
}
}
}
db.close()
I know that's pretty terse, but as I said, see the other thread for a more complete posting. I'm putting it together right now and will post the answer shortly.

Using Groovy in Confluence

I'm new to Groovy and coding in general, but I've come a long way in a very short amount of time. I'm currently working in Confluence to create a tracking tool, which connects to a MySql Database. We've had some great success with this, but have hit a wall with using Groovy and the Run Macro.
Currently, we can use Groovy to populate fields within the Run Macro, which really works well for drop down options, example:
{groovy:output=wiki}
import com.atlassian.renderer.v2.RenderMode
def renderMode = RenderMode.suppress(RenderMode.F_FIRST_PARA)
def getSql = "select * from table where x = y"
def getMacro = '{sql-query:datasource=testdb|table=false} ${getSql} {sql-query}"
def get = subRenderer.render(getMacro, context, renderMode)
def runMacro = """
{run:id=test|autorun=false|replace=name::Name, type::Type:select::${get}|keepRequestParameters = true}
{sql:datasource=testdb|table=false|p1=\$name|p2=\$type}
insert into table1 (name, type) values (?, ?)
{sql}
{run}
"""
out.println runMacro
{groovy}
We've also been able to use Groovy within the Run Macro, example:
enter code here
{run:id=test|autorun=false|replace=name::Name, type::Type:select::${get}|keepRequestParameters = true}
{groovy}
def checkSql = "{select * from table where name = '\name' and type = '\$type'}"
def checkMacro = "{sql-query:datasource=testdb|table=false} ${checkSql} {sql-query}"
def check = subRenderer.render(checkMacro, context, renderMode)
if (check == "")
{
println("This information does not exist.")
} else {
println(checkMacro)
}
{groovy}
{run}
However, we can't seem to get both scenarios to work together, Groovy inside of a Run Macro inside of Groovy.
We need to be able to get the variables out of the Run Macro form so that we can perform other functions, like checking the DB for duplicates before inserting data.
My first thought is to bypass the Run Macro and create a simple from in groovy, but I haven't been too lucky with finding good examples. Can anyone help steer me in the right direction for creating a simple form in Groovy that would replace the Run Macro? Or have suggestions on how to get the rendered variables out of the Run Macro?

Play + Slick: How to do partial model updates?

I am using Play 2.2.x with Slick 2.0 (with MYSQL backend) to write a REST API. I have a User model with bunch of fields like age, name, gender etc. I want to create a route PATCH /users/:id which takes in partial user object (i.e. a subset of the fields of a full user model) in the body and updates the user's info. I am confused how I can achieve this:
How do I use PATCH verb in Play 2.2.x?
What is a generic way to parse the partial user object into an update query to execute in Slick 2.0?I am expecting to execute a single SQL statement e.g. update users set age=?, dob=? where id=?
Disclaimer: I haven't used Slick, so am just going by their documentation about Plain SQL Queries for this.
To answer your first question:
PATCH is just-another HTTP verb in your routes file, so for your example:
PATCH /users/:id controllers.UserController.patchById(id)
Your UserController could then be something like this:
val possibleUserFields = Seq("firstName", "middleName", "lastName", "age")
def patchById(id:String) = Action(parse.json) { request =>
def addClause(fieldName:String) = {
(request.body \ fieldName).asOpt[String].map { fieldValue =>
s"$fieldName=$fieldValue"
}
}
val clauses = possibleUserFields.flatMap ( addClause )
val updateStatement = "update users set " + clauses.mkString(",") + s" where id = $id"
// TODO: Actually make the Slick call, possibly using the 'sqlu' interpolator (see docs)
Ok(s"$updateStatement")
}
What this does:
Defines the list of JSON field names that might be present in the PATCH JSON
Defines an Action that will parse the incoming body as JSON
Iterates over all of the possible field names, testing whether they exist in the incoming JSON
If so, adds a clause of the form fieldname=<newValue> to a list
Builds an SQL update statement, comma-separating each of these clauses as required
I don't know if this is generic enough for you, there's probably a way to get the field names (i.e. the Slick column names) out of Slick, but like I said, I'm not even a Slick user, let alone an expert :-)