I have a table called "KLIJENTI" in MySQL database on ampps localhost server. Table consists of 4 columns: ID_KLIJENTA - int, IME_KLIJENTA - string, PREZIME_KLIJENTA - string and ADRESA_KLIJENTA - string.
I followed tutorials on how to use slick to insert into and read from database, but it does not do what I want.
Here is my application.conf:
scalaTest = {
url = "jdbc:mysql://localhost/Scala1"
user = "root"
password = "mysql"
driver = com.mysql.jdbc.Driver
connectionPool = disabled
keepAliveConnection = true
logStatements = true
}
build.sbt:
name := """project4"""
mainClass in Compile := Some("HelloSlick")
libraryDependencies ++= List(
"com.typesafe.slick" %% "slick" % "3.1.0-RC2",
"org.slf4j" % "slf4j-nop" % "1.7.10",
"com.h2database" % "h2" % "1.4.187",
"org.scalatest" %% "scalatest" % "2.2.4" % "test" ,
"mysql" % "mysql-connector-java" % "5.1.28"
)
fork in run := true
And here is my helloSlick.scala:
import scala.concurrent.{Future, Await}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.Duration
import slick.backend.DatabasePublisher
import slick.driver.MySQLDriver.api._
case class Klijent(ID_KLIJENTA: Option[Int] = None, IME_KLIJENTA: String, PREZIME_KLIJENTA: String, ADRESA_KLIJENTA: String)
class Klijenti(tag: Tag) extends Table[Klijent](tag, "KLIJENTI") {
def ID_KLIJENTA = column[Option[Int]]("ID_KLIJENTA", O.PrimaryKey, O.AutoInc)
def IME_KLIJENTA = column[String]("IME_KLIJENTA")
def PREZIME_KLIJENTA = column[String]("PREZIME_KLIJENTA")
def ADRESA_KLIJENTA = column[String]("ADRESA_KLIJENTA")
def * = (ID_KLIJENTA, IME_KLIJENTA, PREZIME_KLIJENTA , ADRESA_KLIJENTA) <>((Klijent.apply _).tupled, Klijent.unapply)
}
// The main application
object HelloSlick extends App {
val db = Database.forConfig("scalaTest")
val mojiKlijenti = TableQuery[Klijenti]
val q1 = sql"select IME_KLIJENTA from KLIJENTI WHERE ID==1 ".as[String]
val q2 = mojiKlijenti.filter(_.ID_KLIJENTA === 0)
println(q1)
println(q2)
Before I run this code, I already inserted one row into the database through phpmyadmin, so both values q1 and q2 should print out real values but instead of that, I get this:
background log: info: Running HelloSlick
background log: info: slick.jdbc.SQLActionBuilder$$anon$1#56300388
background log: info: Rep(Filter #1636634026)
My apache webserver and mysql are turned on, just in case someone asks..
Why am I not getting the real values?
EDIT:> Later on, I tried to execute following command mojiKlijenti += Klijent(Some(5),"Name","Surname","Address") and again, it compiler successfully but when I check my database in phpmyadmin, no record is added..
You are not getting the real values because you need to convert the Query into an Action by calling its result method and execute it.
val action = q2.result
val results = db.run( action)
results.foreach( println )
Related
I've tried to implement this pipeline in my spider.
After installing the necessary dependencies I am able to run the spider without any errors but for some reason it doesn't write to my database.
I'm pretty sure there is something going wrong with connecting to the database. When I give in a wrong password, I still don't get any error.
When the spider scraped all the data, it needs a few minutes before it starts dumping the stats.
2017-08-31 13:17:12 [scrapy] INFO: Closing spider (finished)
2017-08-31 13:17:12 [scrapy] INFO: Stored csv feed (27 items) in: test.csv
2017-08-31 13:24:46 [scrapy] INFO: Dumping Scrapy stats:
Pipeline:
import MySQLdb.cursors
from twisted.enterprise import adbapi
from scrapy.xlib.pydispatch import dispatcher
from scrapy import signals
from scrapy.utils.project import get_project_settings
from scrapy import log
SETTINGS = {}
SETTINGS['DB_HOST'] = 'mysql.domain.com'
SETTINGS['DB_USER'] = 'username'
SETTINGS['DB_PASSWD'] = 'password'
SETTINGS['DB_PORT'] = 3306
SETTINGS['DB_DB'] = 'database_name'
class MySQLPipeline(object):
#classmethod
def from_crawler(cls, crawler):
return cls(crawler.stats)
def __init__(self, stats):
print "init"
#Instantiate DB
self.dbpool = adbapi.ConnectionPool ('MySQLdb',
host=SETTINGS['DB_HOST'],
user=SETTINGS['DB_USER'],
passwd=SETTINGS['DB_PASSWD'],
port=SETTINGS['DB_PORT'],
db=SETTINGS['DB_DB'],
charset='utf8',
use_unicode = True,
cursorclass=MySQLdb.cursors.DictCursor
)
self.stats = stats
dispatcher.connect(self.spider_closed, signals.spider_closed)
def spider_closed(self, spider):
print "close"
""" Cleanup function, called after crawing has finished to close open
objects.
Close ConnectionPool. """
self.dbpool.close()
def process_item(self, item, spider):
print "process"
query = self.dbpool.runInteraction(self._insert_record, item)
query.addErrback(self._handle_error)
return item
def _insert_record(self, tx, item):
print "insert"
result = tx.execute(
" INSERT INTO matches(type,home,away,home_score,away_score) VALUES (soccer,"+item["home"]+","+item["away"]+","+item["score"].explode("-")[0]+","+item["score"].explode("-")[1]+")"
)
if result > 0:
self.stats.inc_value('database/items_added')
def _handle_error(self, e):
print "error"
log.err(e)
Spider:
import scrapy
import dateparser
from crawling.items import KNVBItem
class KNVBspider(scrapy.Spider):
name = "knvb"
start_urls = [
'http://www.knvb.nl/competities/eredivisie/uitslagen',
]
custom_settings = {
'ITEM_PIPELINES': {
'crawling.pipelines.MySQLPipeline': 301,
}
}
def parse(self, response):
# www.knvb.nl/competities/eredivisie/uitslagen
for row in response.xpath('//div[#class="table"]'):
for div in row.xpath('./div[#class="row"]'):
match = KNVBItem()
match['home'] = div.xpath('./div[#class="value home"]/div[#class="team"]/text()').extract_first()
match['away'] = div.xpath('./div[#class="value away"]/div[#class="team"]/text()').extract_first()
match['score'] = div.xpath('./div[#class="value center"]/text()').extract_first()
match['date'] = dateparser.parse(div.xpath('./preceding-sibling::div[#class="header"]/span/span/text()').extract_first(), languages=['nl']).strftime("%d-%m-%Y")
yield match
If there are better pipelines available to do what I'm trying to achieve that'd be welcome as well. Thanks!
Update:
With the link provided in the accepted answer I eventually got to this function that's working (and thus solved my problem):
def process_item(self, item, spider):
print "process"
query = self.dbpool.runInteraction(self._insert_record, item)
query.addErrback(self._handle_error)
query.addBoth(lambda _: item)
return query
Take a look at this for how to use adbapi with MySQL for saving scraped items. Note the difference in your process_item and their process_item method implementation. While you return the item immediately, they return Deferred object which is the result of runInteraction method and which returns the item upon its completion. I think this is the reason your _insert_record never gets called.
If you can see the insert in your output that's already a good sign.
I'd rewrite the insert function this way:
def _insert_record(self, tx, item):
print "insert"
raw_sql = "INSERT INTO matches(type,home,away,home_score,away_score) VALUES ('%s', '%s', '%s', '%s', '%s')"
sql = raw_sql % ('soccer', item['home'], item['away'], item['score'].explode('-')[0], item['score'].explode('-')[1])
print sql
result = tx.execute(sql)
if result > 0:
self.stats.inc_value('database/items_added')
It allows you to debug the sql you're using. In you version you're not wrapping the string in ' which is a syntax error in mysql.
I'm not sure about your last values (score) so I treated them as strings.
WHAT I DID:
I am trying to implement oauth2 provider using Play framework. I am using the "scala-oauth2-provider" sample to do this. "https://github.com/nulab/scala-oauth2-provider"
I have listed the version that I used in my application:
Play -- 2.5
database -- Mysql 5.1.22
ISSUE:
AuthCode.scala:19: could not find implicit value for parameter tm:
scala.slick.ast.TypedType[org.joda.time.DateTime] [error] def
createdAt = columnDateTime
Code Snippet:
import java.util.UUID
import org.joda.time.DateTime
import scala.slick.driver.MySQLDriver.simple._
case class AuthCode(authorizationCode: String, userGuid: UUID, redirectUri: Option[String], createdAt: DateTime, scope: Option[String], clientId: Option[String], expiresIn: Int)
class AuthCodes(tag: Tag) extends Table[AuthCode](tag, "auth_codes") {
def authorizationCode = column[String]("authorization_code", O.PrimaryKey)
def userGuid = column[UUID]("user_guid")
def redirectUri = column[Option[String]]("redirect_uri")
def createdAt = column[DateTime]("created_at")
def scope = column[Option[String]]("scope")
def clientId = column[Option[String]]("client_id")
def expiresIn = column[Int]("expires_in")
def * = (authorizationCode, userGuid, redirectUri, createdAt, scope, clientId, expiresIn) <> (AuthCode.tupled, AuthCode.unapply)
}
How do I fix this issue? Can anyone help me to fix this issue?
Note: I have also tried the solution https://stackoverflow.com/a/22578950/1584121. But I am getting same issue :(
I didn't look into the sample project, but I used the solution available through https://github.com/tototoshi/slick-joda-mapper. Hopefully you can apply this to your code. I am using Scala 2.11.8 with Slick 3.1.1, and therefore use slick-joda-mapper 2.2.0. If you are using different versions of Scala and/or Slick, you might have to select a different version of slick-joda-mapper.
First add the required dependencies to your build.sbt:
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "3.1.1",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"com.github.tototoshi" %% "slick-joda-mapper" % "2.2.0",
"joda-time" % "joda-time" % "2.7",
"org.joda" % "joda-convert" % "1.7"
)
In your Scala source file, use the following imports:
import slick.driver.MySQLDriver.api._
import scala.concurrent.ExecutionContext.Implicits.global
import com.github.tototoshi.slick.MySQLJodaSupport._
import org.joda.time.DateTime
You can then use a DateTime object simply in Slick queries.
I'm currently trying to create a table using slick and I'm baffled as to what import I'm missing as the examples I've seen don't seem to have a relevant looking import in them.
Currently the column, question mark and the O's are all unresolved.
Could someone let me know what I'm doing wrong please?
Here is my table class:
package com.grimey.tabledefinitions
import slick.driver.MySQLDriver.api._
import com.grimey.staticpage.StaticPage
import slick.lifted.Tag
import slick.model.Table
class StaticPageDef(tag: Tag) extends Table[StaticPage](tag, "static_page") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def pageType = column[String]("page_type")
def contentHtml = column[String]("content_html")
def * = (id.?, pageType, contentHtml) <>(StaticPage, StaticPage.unapply _)
}
And here is my build.sbt:
name := """grimey-cms"""
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"mysql" % "mysql-connector-java" % "5.1.38",
"com.typesafe.play" %% "play-slick" % "2.0.0",
"com.typesafe.play" %% "play-slick-evolutions" % "2.0.0"
)
resolvers += "scalaz-bintray" at "http://dl.bintray.com/scalaz/releases"
fork in run := true
And finally, here is the case class I'm using for the table:
package com.grimey.staticpage
import java.time.LocalDateTime
case class StaticPage(id: Long, htmlContent: String, pageType: String,
created: LocalDateTime, updated: LocalDateTime)
I bet it's something really silly :)
The O object is from the table and it varies driver to driver. Some drivers may not support certain column options supported by others. Therefore you will need to import the column options that are specific to your database - MySQL in this case:
import slick.driver.MySQLDriver.api._
You may check this full tutorial on how to use Play + Slick + MySQL: http://pedrorijo.com/blog/play-slick/
Or you may just navigate through the code: https://github.com/pedrorijo91/play-slick3-steps
I'm writing a veery simple scala script to connect to Mysql using slick 3.
My build.sbt looks like this:
name := "slick_sandbox"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % "3.0.3",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"mysql" % "mysql-connector-java" % "5.1.6"
)
application.conf:
Drivder is an intentional mistake; also, I did not provide a db username or password!
mysqldb = {
url = "jdbc:mysql://localhost/slickdb"
driver = com.mysql.jdbc.Drivder
connectionPool = disabled
keepAliveConnection = true
}
Main.scala
import slick.driver.MySQLDriver.api._
import scala.concurrent.ExecutionContext.Implicits.global
object Main {
def main(args: Array[String]) {
// test to see this function is being run; it IS
println("foobar")
// I expected an error here due to the intentional
// mistake I've inserted into application.conf
// I made sure the conf file is getting read; if I change mysqldb
// to some other string, I get correctly warned it is not a
// valid key
val db = Database.forConfig("mysqldb")
val q = sql"select u.name from users ".as[String]
db.run(q).map{ res=>
println(res)
}
}
}
It compiles OK. Now this is the result I see when I run sbt run on the terminal:
felipe#felipe-XPS-8300:~/slick_sandbox$ sbt run
[info] Loading project definition from /home/felipe/slick_sandbox/project
[info] Set current project to slick_sandbox (in build file:/home/felipe/slick_sandbox/)
[info] Compiling 1 Scala source to /home/felipe/slick_sandbox/target/scala-2.11/classes...
[info] Running Main
foobar
[success] Total time: 5 s, completed Sep 17, 2015 3:29:39 AM
Everything looks deceptively OK; even though I explicitly ran the query on a database that doesn't exist, slick went ahead as if nothing has happened.
What am I missing here?
Slick runs queries asynchronously. So it just didn't have enough time to execute it. In your case you have to wait for result.
object Main {
def main(args: Array[String]) {
println("foobar")
val db = Database.forConfig("mysqldb")
val q = sql"select u.name from users ".as[String]
Await.result(
db.run(q).map{ res=>
println(res)
}, Duration.Inf)
}
}
The methods in ADSRegistrationMap are used to save and retrieve the document from MongoDB. ObjectId is created during initialization. I have to do the same to load Registration from Json that is part of POST body, so I thought I could just add ADSRegistrationProtocol object to do that. It fails with compilation error. Any idea on how to fix it or do this better?
package model
import spray.json._
import DefaultJsonProtocol._
import com.mongodb.casbah.Imports._
import org.bson.types.ObjectId
import com.mongodb.DBObject
import com.mongodb.casbah.commons.{MongoDBList, MongoDBObject}
case class Registration(
system: String,
identity: String,
id: ObjectId = new ObjectId())
object RegistrationProtocol extends DefaultJsonProtocol {
implicit val registrationFormat = jsonFormat2(Registration)
}
object RegistrationMap {
def toBson(registration: Registration): DBObject = {
MongoDBObject(
"system" -> registration.system,
"identity" -> registration.identity,
"_id" -> registration.id
)
}
def fromBson(o: DBObject): Registration = {
Registration(
system = o.as[String]("system"),
identity = o.as[String]("identity"),
id = o.as[ObjectId]("_id")
)
}
}
Compilation Error:
[error] /model/Registration.scala:20: type mismatch;
[error] found : model.Registration.type
[error] required: (?, ?) => ?
[error] Note: implicit value registrationFormat is not applicable here because it comes after the application point and it lacks an explicit result type
[error] implicit val registrationFormat = jsonFormat2(Registration)
[error] ^
[error] one error found
[error] (compile:compile) Compilation failed
Updated ObjectId to String and jsonFormat2 to jsonFormat3 to fix the compilation error.
case class Registration(
system: String,
identity: String,
id: String = (new ObjectId()).toString())
object RegistrationProtocol extends DefaultJsonProtocol {
implicit val registrationFormat = jsonFormat3(Registration)
}
Getting runtime error now when converting body of POST request to the Registration object. Any idea?
val route: Route = {
pathPrefix("registrations") {
pathEnd {
post {
entity(as[Registration]) { registration =>
Here is what is in build.sbt
scalaVersion := "2.10.4"
scalacOptions ++= Seq("-feature")
val akkaVersion = "2.3.8"
val sprayVersion = "1.3.1"
resolvers += "spray" at "http://repo.spray.io/"
resolvers += "Sonatype releases" at "https://oss.sonatype.org/content/repositories/releases"
// Main dependencies
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion,
"com.typesafe.akka" %% "akka-camel" % akkaVersion,
"io.spray" % "spray-can" % sprayVersion,
"io.spray" % "spray-routing" % sprayVersion,
"io.spray" % "spray-client" % sprayVersion,
"io.spray" %% "spray-json" % sprayVersion,
"com.typesafe" % "config" % "1.2.1",
"org.apache.activemq" % "activemq-camel" % "5.8.0",
"ch.qos.logback" % "logback-classic" % "1.1.2",
"org.mongodb" %% "casbah" % "2.7.4"
)
Error:
12:33:03.477 [admcore-microservice-system-akka.actor.default-dispatcher-3] DEBUG s.can.server.HttpServerConnection - Dispatching POST request to http://localhost:8878/api/v1/adsregistrations to handler Actor[akka://admcore-microservice-system/system/IO-TCP/selectors/$a/1#-1156351415]
Uncaught error from thread [admcore-microservice-system-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[admcore-microservice-system]
java.lang.NoSuchMethodError: spray.json.JsonParser$.apply(Ljava/lang/String;)Lspray/json/JsValue;
at spray.httpx.SprayJsonSupport$$anonfun$sprayJsonUnmarshaller$1.applyOrElse(SprayJsonSupport.scala:36)
at spray.httpx.SprayJsonSupport$$anonfun$sprayJsonUnmarshaller$1.applyOrElse(SprayJsonSupport.scala:34)
To avoid any issue, I would define the Register class (which seems a data model) as follows
case class Register(system: String, identity: String, id: String)
That's because it makes more sense to me having an id field as String rather than as BSON ObjectId (I'm used to datamodel which don't depend on 3rd party libraries).
Therefore, the right SprayJson protocol would make use of jsonFormat3 rather than jsonFormat2:
object RegistrationProtocol extends DefaultJsonProtocol {
implicit val registrationFormat = jsonFormat3(Registration)
}
And that would solve any kind of JSON serialization issue.
Finally, your toBson and fromBson converters would be:
def toBson(r: Registration): DBObject = {
MongoDBObject(
"system" -> r.system,
"identity" -> r.identity,
"_id" -> new ObjectId(r.id)
)
}
and
def fromBson(o: DBObject): Registration = {
Registration(
system = o.as[String]("system"),
identity = o.as[String]("identity"),
id = o.as[ObjectId]("_id").toString
)
}
A that's where the BSON ObjectId is being used: much closer to the MongoDB dependant logic.