In my project I need to collect data, process/Analyze data and published it.
I am using WSO2 DAS 3.0.0 for the same.
In order to achieve this I have followed below steps:-
Created event stream(define attribute).
Created Event receiver. (HTTP,SOAP)
Created event publisher (store event in My sql but getting error in SQL server see below).
My question is as below:-
Is there any other way to store persisted event in Database. Can I store event in database using Main->stream->persist event. Can I use this to store event in Database or I need to use Main->publishers. Select "output event adapter type"- RDBMS to store the event?
Can I use SQL server 2008 to store the event in SQL server 2008? I am getting below error while storing data into SQL server 2008.
Cannot Execute Create Table Query. There is already an object named 'tbl_City' in the database. Hence Event is dropped.
TID[-1234] [DAS] [2015-11-09 12:28:06,265] ERROR {org.wso2.carbon.event.output.adapter.rdbms.RDBMSEventAdapter} - Cannot Execute Create Table Query. There is already an object named 'tbl_City' in the database. Hence Event is dropped. org.wso2.carbon.event.output.adapter.rdbms.RDBMSEventAdapter.createTableIfNotExist(RDBMSEventAdapter.java:416) org.wso2.carbon.event.output.adapter.rdbms.RDBMSEventAdapter.executeProcessActions(RDBMSEventAdapter.java:290) org.wso2.carbon.event.output.adapter.rdbms.RDBMSEventAdapter.publish(RDBMSEventAdapter.java:134) org.wso2.carbon.event.output.adapter.core.internal.OutputAdapterRuntime.publish(OutputAdapterRuntime.java:62) org.wso2.carbon.event.output.adapter.core.internal.CarbonOutputEventAdapterService.publish(CarbonOutputEventAdapterService.java:143) org.wso2.carbon.event.publisher.core.internal.EventPublisher.process(EventPublisher.java:361) org.wso2.carbon.event.publisher.core.internal.EventPublisher.sendEvent(EventPublisher.java:209) org.wso2.carbon.event.publisher.core.internal.EventPublisher.consumeEvent(EventPublisher.java:275) org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:142) org.wso2.carbon.event.receiver.core.internal.management.InputEventDispatcher.onEvent(InputEventDispatcher.java:27) org.wso2.carbon.event.receiver.core.internal.EventReceiver.sendEvent(EventReceiver.java:259) org.wso2.carbon.event.receiver.core.internal.EventReceiver.processTypedEvent(EventReceiver.java:237) org.wso2.carbon.event.receiver.core.internal.EventReceiver$TypedEventSubscription.onEvent(EventReceiver.java:317) org.wso2.carbon.event.input.adapter.core.internal.InputAdapterRuntime.onEvent(InputAdapterRuntime.java:110) org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet$HTTPRequestProcessor.run(HTTPMessageServlet.java:210) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745)
Please confirm.
Seems like you have confused event publishers with Data publishers.
Events publishers are used to publish results to external systems for taking further actions. Event publishers provide the capability to send event notifications and alerts from WSO2 DAS to external systems.
For your use case, what you have to
do is publishing data to DAS. There are couple of ways of doing that. Please have a look on [1] for more information.
You can also follow the DAS quick start guide [2] to get a better understanding about the concepts.
About the question on using SQL server 2008, yes DAS support Microsoft SQL server.
[1] https://docs.wso2.com/display/DAS300/Publishing+Data+to+DAS
[2] https://docs.wso2.com/display/DAS300/Quick+Start+Guide
Thanks for your reply.
Yes you are correct. I got confused between event/data publisher. I dont want to publish my event I want it to be stored in the SQL server.
Please go through my system flow as below:-
An external system will be pushing data into DAS. That data I need to store in SQL server and need to processed for further analysis.
I did the below steps for it.
Created event stream. Define 5 attributes. :- Done
Created Event receiver. Exposed HTTP endpoint.:- Done
I need to store this data into the SQL server 2008. - Pending
I can publishes below sample data to DAS.:-
"<events>
<event>
<payloadData>
<queueName>data3</queueName>
<agentsOnThreshold>100</agentsOnThreshold>
<agentsAvailThreshold>45</agentsAvailThreshold>
<callsWtngThreshold>4</callsWtngThreshold>
<avgWaitInSecThreshold>100</avgWaitInSecThreshold>
<oldestCallInSecThreshold>45</oldestCallInSecThreshold>
</payloadData>
</event>`enter code here`
"
So my question is how to store above data into SQL server which I received in DAS (event receiver).
In order to persist the data that is being received by DAS for a particular stream, you need to enable data persistence for it. Please follow [1] and persist the stream if you haven't done that yet.
You can verify the DAS data recieving functionality by sending a sample event to DAS by using DAS inbuilt event simulation feature [2].
To make sure whether data is persisted in Event Store, go to the Data Exporer [3] and explore your published data.
[1] https://docs.wso2.com/display/DAS300/Persisting+Data+for+Batch+Analytics
[2] https://docs.wso2.com/display/DAS300/Publishing+Data+Using+Event+Simulation
[3] https://docs.wso2.com/display/DAS300/Data+Explorer
Thanks for the links and information it really help me a lot.
I am able to save the event into the SQL server 2008.
I just wanted to add what steps I have followed:-
Open \repository\conf\datasources\master-datasources.xml
change the datasource tag. Change URL, username,password, driveclassname.
Follow below link for the details steps:-
https://docs.wso2.com/display/DAS300/Setting+up+Microsoft+SQL
Open \repository\conf\datasources\analytics-datasources.xml
Change the configuration settings. Change URL, username,password, driveclassname.
e.g.
"<datasource>
<name>WSO2_ANALYTICS_FS_DB</name>
<description>The datasource used for analytics file system</description>
<definition type="RDBMS">
<configuration>
jdbc:sqlserver://localhost:1433;databaseName=testDAS
sa
******
driverClassName>com.microsoft.sqlserver.jdbc.SQLServerDriver
50
60000
5
true
SELECT 1
30000
"
Make similar changes to the "WSO2_ANALYTICS_EVENT_STORE_DB", "WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB".
Create an event and check in SQL server table whether it get capture. In my case SQL table is created with name "[ANX___7Lk3JrzI_]".
Thanks.
Related
I am trying to connect my Soffid 3 server with our custom web application named Schrift. I am using а JSON REST Web Services Connector for this purpose. I added REST Web service plugin and then configured an agent with JSON/XML/SOAP Rest webservice type.
Loading of objects is working fine. My REST connector connects to the web service successfully and gets data of the accounts.
The problem is when I am trying to update some data (for example, I am trying to lock an account), nothing happens. And unfortunately I don't know what should be happening. When should REST connector send updated data to the managed system and in which way? I didn't find any log entries saying that REST connector was trying to update an object on managed system. Maybe I did smth wrong or missed something.
I would appreciate for any help. I can post any conf or log details if you need.
Update#1
(I did some investigation after the first answer)
I checked the agent settings: Read only and Manual account creation are set to no
The account was set to unmanaged type, but I succeeded in changing its type to shared and then to single without getting an error. Now it is set to single
The task queue is empty.
Also I've checked that update method is present and update properties are set correctly. updateParams is not set (it means that all attributes should be sent to the managed system).
But when I change status of the account (from Enable to Disable), nothing happens.
In the console log I can see only these lines
14-Sep-2021 13:26:29.708 INFO [BPM-Scheduler:192.168.7.121:1] com.soffid.iam.bpm.job.JobExecutorThread.run No job to execute
When I manually run the task Analize impact for changes on Schrift, Execution log shows
Changes detected for accounts
=============================
NO CHANGE DETECTED
Changes detected for roles
=============================
NO CHANGE DETECTED
Update#2
After many attempts I made some progress. Now when I make some changes in the account, the task named UpdateAccount baklykov#irf.com.ua#Schrift appears, but runs with an error.
At first it was 415 Unsupported Media Type error as I wrote in comments, but now it looks a little different
Throws exception updating object : Extensible object [type = account]
EmployeeEmail: baklykov#irf.com.ua
IsLockedOut: true (log truncated) ...
caused by Unexpected response, Content-Type: null
Update#3
I found out that soffid's request for updating the object was in improper format (all the parameters were passed in the html request instead of putting them in json body)
After researching I found a method's property called Encoding and set it to application/json value.
Now the parameters are passed in json body (that's what I need), but now the problem is that soffid puts all the parameters in json body, including the key parameter by which the object for updating should be determined. My guess this is the reason why the object in the target system is still not updated.
In other words my application expects a request like this:
https://myapp.mysite.com/api/v1/Soffid/Employees?EmployeeEmail=baklykov%40irf.com.ua :
{"EmployeeLastName":"Baklykov","EmployeeFirstName":"Ivan"}
but Soffid sends this:
https://myapp.mysite.com/api/v1/Soffid/Employees:
{"EmployeeLastName":"Baklykov","EmployeeFirstName":"Ivan","EmployeeEmail":"baklykov#irf.com.ua"}
The system should have created a UpdateAccount task in the task queue. Please, verify:
The task engine is in automatic mode. In read-only or manual mode, no task will be created.
If you are updating an account, check the account is not set as unmanaged. In that case, no tasks is created.
Finally, verify the task queue has not held the task up.
Have you checked the engine mode? Look at Main Menu > Administration > Configure Soffid > Integration engine > Smart engine settings
It should be set to automatic.
Hi i have problem with Presta WebService. I'm using prestashop with node (nestJS). I Created a connection with MySql database (Presta DB) with SSH tunelling. It works corectly, i can do any query, and SQL returns "OK statuses". I checked it by ssh client, (queries in temrinal) and data is updated in DB. Problem is when i want to select this data by PrestaWebService, when i pick data presta returns "old" propably cached data. On presta Admin i have ceching "off". After manually reset cache by admin panel button keep presta returns me 'old" data. Anyone know whats going on? :D
I found a problem. I used incorrect table. I queried ps_product but should ps_product_shop.
Is there an equivalent of PostgresQL's notify and listen in MySQL? Basically, I need to listen to triggers in my Java application server.
Ok, so what I found is that you can make UDF functions in mysql that can do anything but need to be written in C/C++. They can be then called from triggers on updates in database and notify your application when update happened. I saw that there are some security concerns. I did not use it myself but from what I can see it looks like something that could accomplish what you want to do and more.
http://dev.mysql.com/doc/refman/5.6/en/adding-udf.html
The github project mysql-notification provides a MySQL user defined function MySQLNotification() as a plugin to MySQL that will send notification events via a socket interface. This project includes a sample NodeJS test server that receives the notification events that could be adapted for Java or any other socket service.
Example use:
$ DELIMITER ##
$ CREATE TRIGGER <triggerName> AFTER INSERT ON <table>
FOR EACH ROW
BEGIN
SELECT MySQLNotification(NEW.id, 2) INTO #x;
END##
Project includes full source code and installation instructions for OSX and Linux. License is GNU v3.
No, there aren't any built-in functions like these yet.
You need to "ping" (every 1-5 seconds) database with selecting with premade flag like "read" 0/1. After
SELECT * FROM mytable WHERE read = 0
update it with read = 1
I needed to do this, so I designed my application to send the update notices itself.
E.g.
--Scenario--
User A is looking at record 1
User B saves an update to record 1 while User A has it open.
Process:
I wrote my own socket server as a Windows Service. I designed a que like system which is basically,
EntityType EntityID NoticeType
Where the EntityType is the type of Poco in my data layer that needs to send out notices, EntityID is the primary key value of the row that changed in sql (the values of the poco), and NoticeType is 1 Updated, 2 Inserted, and 3 Deleted.
The socket server accepts connections from the server side application code on a secure connection "meaning client side code cannot make requests designed to be sent by the server side application code"
The socket server accepts a message like
900 1 1023 1
Which would mean the server needs to notify concerned client connections that Entity Type 1 "Person" with ID 1023 was Updated.
The server knows what users need to be notified because when User's look at a record, they are registered in the socket server as having an interest in the record and the record's ID which is done by the web socket code in the client side javascript.
Record 1 is a POCO in my app code that has an IsNew and IsDirty field. "Using EntityFrameWork6 and MySql" If UserB's save caused an actual change (and not just saving existing data) IsDirty will be true on the postback on UserB's POCO.
The application code see's the record is dirty then notifies the socket server with a server side sent socket "which will be allowed" that says Entity 1 with ID 1023 was Updated.
The socket server sees it, and puts it in the que.
Being .Net, I have a class for concerned users that uses the same pocos from the data layer running in the Socket Server window service. I use linq to select users who are working with an entity matching the entity type and primary key id of the entity in the que.
It then loops through those users and sends them a socket like
901 1 1023 1 letting them know the entity was updated.
The javascript in the client side receives it causing users B's page to do an ajax postback on Record 1, But what happens with UserA's is different.
If user A was in the process of making a change, they will get a pop up to show them what changed, and what their new value will be if they click save and asks them which change they want to keep. If UserA doesn't have a change it does an ajax postback with a notification bar at the top that says "Record Change: Refreshed Automatically" that expires after a few seconds.
The Cons to this,
1. It's very complex
2. It won't catch insert/update/delete operations from outside of the application.
In my case, 2 won't happen and if 2 does happen it's by myself or another dev who knows how to manually create the notify que requests "building an admin page for that".
You can use https://maxwells-daemon.io to do so.
It is based on mysql bin logs, when changes in database is occurred it will send json message with updates to kafka, rabbitmq or other streaming platforms
I have a rather complex and large database with about 3000+ objects (tables/triggers/sps combined). I inherited this DB and restructuring it is probably 3-4 years away.
meanwhile, I need to implement a pub sub feature for any insert/update/delete on these tables. Given number of tables and existing queries probably query notification (and SQL Dependency) will not work. What I am looking for is a way to push the changes (what changed in table - like records PK and table name) on the service broker so I can use external activator to then retrieve change,and then use my custom pub sub from that point onwards.
I have pretty much all the ducks lined up except for the way to push change notification on service broker.
Any help/pointers are appreciated.
Thanks.
N M
PS. I did look around for similar postings and did come across a few however, MSDN articles they referred to seem to have all removed - not sure what's going on on MSDN site.
For external activator look at Microsoft SQL Server 2008 Feature Pack- "Microsoft SQL Server 2008 R2 Service Broker External Activator".
For console application (that processes messages) great idea is to drop an eye in codeplex. There is good examples.
To put event notification (notifications, that will be used by external activator service) code looks something like this:
Create Queue ExternalActivatorQueue;
Create Service ExternalActivatorService On Queue ExternalActivatorQueue
([http://schemas.microsoft.com/SQL/Notifications/PostEventNotification])
Create Event Notification NotifyExternalActivator
On Queue dbo.ProcessQueue
For QUEUE_ACTIVATION
To Service 'ExternalActivatorService', 'current database'
To send message in the queue:
Declare #h UniqueIdentifier;
Declare #x xml = '<tag/>';
Begin Dialog Conversation #h
From Service MyTableService
To Service 'ProcessService'
With Encryption = OFF;
Send On Conversation #h(#x)
All steps i done to make it work is here, but just in Latvian :). There actually is almost what you need (trigger that sends messages when data are inserted in table..).
I want to execute a job when ever a file is dropped into a particular folder.
I found some articles that showed me how I can do it on SQL Server.
I created a alert type: WMI Event Alert
For the name space its the SQL instance which comes automatically as \\.\root\Microsoft\SqlServer\ServerEvents\MSSQLSERVER
On the Query section - I wrote the below query ,
SELECT *
FROM __InstanceCreationEvent
WITHIN 1
WHERE TargetInstance ISA 'CIM_DataFile'
AND TargetInstance.Name = ‘c:\\TestFolder\’ `
The error message returned is:
Cannot create new alert.
ADDITIONAL INFORMATION:
Create failed for Alert 'AlertTest'. (Microsoft.SqlServer.Smo)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=10.50.2425.0+((KJ_PCU_Main).110406-2044+)&EvtSrc=Microsoft.SqlServer.Management.Smo.ExceptionTemplates.FailedOperationExceptionText&EvtID=Create+Alert&LinkId=20476
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
SQLServerAgent Error: WMI error: 0x80041058
The #wmi_query could not be executed in the #wmi_namespace provided. Verify that an event class selected in the query exists in the namespace and that the query has the correct syntax. (Microsoft SQL Server, Error: 22022)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=10.50.2425&EvtSrc=MSSQLServer&EvtID=22022&LinkId=20476
Please may you advise if my Query is correct and if there is anything else I need to check?
Thanks a lot in advance.
You are using the wrong namespace, the CIM_DataFile WMI class is part of the \root\CIMV2 namespace and not of \root\Microsoft\SqlServer\ServerEvents\MSSQLSERVER
In this case, the answer from RRUZ is correct. However there are other possible causes of this error message:
The #wmi_query could not be executed in the #wmi_namespace provided.
One possible reason is the account that runs the Windows service "Windows Management Instrumentation" is disabled as a SQL login. (If you are running SQL 2012+, look for the login 'NT SERVICE\winmgmt').
(Source: Blog by 'rahmanagoro' )
Edit 2020-05-29: I've made a more comprehensive answer to this question on the DBA forum.
Another possible fix is to restart the "Windows Management Instrumentation" service. No idea what leads to the problem, but restarting the service fixes it. I've seen this twice, both times on Windows Server 2008 R2 Standard Edition x64.
Edit 2020-05-29: I've made a more comprehensive answer to this question on the DBA forum.