We have a few thousand companies in Exact Online from which a certain percentage runs their own accounting and has their own license. However, there is a daily changing group of companies that are behind with their payments to Exact and therefore their companies are blocked.
For all companies we run Invantive Data Replicator to replicate all Exact Online companies into a SQL Server datawarehouse for analytical reporting and continuous monitoring.
In the SystemDivisions table, the state of such a blocked company remains at 1 (Active). It does not change to 2 (Archive) or 0 (upcoming). Nor is there any enddate set in the past.
However, when the XML or REST APIs are used through a query from Invantive SQL or directly from Python on such a blocked company there are lot of fuzzy error messages.
Currently we have to open each company which had an error during replication individually each day and check whether a block by Exact is causing the error and for what reason.
It seems that there is no way to retrieve the list of blocked companies.
Is there an alternative?
Although it is not supported and disadviced, you can access a limited number of screens in Exact Online using native requests. It is rumoured that this is not possible for all screens.
However, you are lucky. The blocking status of a company can be requested using the following queries:
insert into NativePlatformScalarRequests(url, orig_system_group)
select /*+ ods(false) */ 'https://start.exactonline.nl/docs/SysAccessBlocked.aspx?_Division_=' || code
, 'BLOCK-DIV-CHECK-' || code
from systemdivisions
create or replace table currentlyblockeddivisions#inmemorystorage
as
select blockingstatus
, divisioncode
from ( select regexp_replace(result, '.*<table class="WizardSectionHeader" style="width:100%;"><tr><th colspan="2">([^<]*)</th>.*', '$1', 1, 0, 'n') blockingstatus
, replace(orig_system_group, 'BLOCK-DIV-CHECK-', '') divisioncode
from NativePlatformScalarRequests
where orig_system_group like 'BLOCK-DIV-CHECK-%'
)
where blockingstatus not like '%: Onbekend%'
Please note that the hyperlink with '.nl' needs to be replaced when you run on a different country. The same holds for searching on the Dutch term 'Onbekend' ('Unknown' in english).
This query runs several thousand of HTTP request, each requesting the screen with the blocking status of a company. However, when the company is not blocked, the screen reports back a reason of 'Unknown'.
These companies with 'Unknown' reason are probably not blocked. The rest is.
Related
We are running a custom app on Invantive Data Access Point which adds business functionality to Exact Online. For billing purposes, we would like to somehow register actual use of the software as defined in business terms instead of memory used, CPU, SQL statements executed, etc.
We do not yet have custom tables and I would like to keep it that way, so the whole state is kept in memory and in Exact Online only. So "insert into mytable#sqlserver..." is not an option. Neither does Exact Online offer the possibility to create custom tables as with Salesforce.
How can we somehow register billable events, such as "Performed an upload of 8 bank transactions" under this condition?
For billing purposes, you can lift along on the Customer Service infrastructure, which is similar to functionality offered by AWS or Apple for this purpose in their eco system. The "table" which stores the billing events like a Call Detail Record of a PBX is managed by Customer Service infrastructure.
There are two options:
Your apps use the default audit and license event registrations like "User logged on", "First use of partition #xyz", etc. each with a specific message code like 'itgenlic125'.
Your apps define their own event types like "Performed an upload of bank transactions", with a message code 'mybillingmessagecode123' and the number '8' as quantity in the natural key.
The first option is automatically and always done. These data is also used to manage resource consumption and detect runaways.
The second option is best done using Invantive SQL with the data dictionary table "auditevents". All records inserted into auditevents are automatically asynchronously forwarded to Customer Service. To see the current register audit events since start of application:
select *
from auditevents#datadictionary
where:
occurrence_date: when it happened.
logging_level: always "Audit".
message_code: code identifying the type of event.
data_container_d: ID of the data container, used with distributed SQL transactions.
partition: partition within the data container for platforms such as Exact Online or Microsoft SQL Server which store multiple databases under one customer/instance.
session_id: ID of the session.
user_message: actual text.
last_nk: last used natural key
application_name: name of the appplication.
application_user: user as known to the application.
gui_action: action within the GUI.
And some auditing and licensing information fields.
To register a custom event:
insert into auditevents#datadictionary select * from auditevents#datadictionary
Only some fields can be provided; the rest are automatically determined:
message_code
user_message
last_natural_key
application_name
application_user
gui_action
gui_module
partition
provider_name
reference_key
reference_table_code
session_id
To receive the billing events yourself from the infrastructure, you will need to access the Customer Service APIs or have them automatically forwarded to mail, Slack, RocketChat or Mattermost channel.
A sample SQL:
insert into auditevents#datadictionary
( message_code
, user_message
, last_natural_key
, application_name
, gui_action
, gui_module
, reference_key
, reference_table_code
, partition
)
select 'xxmycode001' message_code
, 'Processed PayPal payments in Exact Online for ' || divisionlabel user_message
, 'today' last_natural_key
, 'PayPalProcessor' application_name
, 'xx-my-paypal-processor-step-2' gui_action
, 'xx-my-payal-processor' gui_module
, clr_id reference_key
, 'clr' reference_table_code
, division partition
from settings#inmemorystorage
This query creates an export for UPS from the deliveries history:
select 'key'
, ACC.Name
, CON.FullName
, CON.Phone
, ADR.AddressLine1
, ADR.AddressLine2
, ADR.AddressLine3
, ACC.Postcode
, ADR.City
, ADR.Country
, ACC.Code
, DEL.DeliveryNumber
, CON.Email
, case
when CON.Email is not null
then 'Y'
else 'N'
end
Ship_Not_Option
, 'Y' Ship_Not
, 'ABCDEFG' Description_Goods
, '1' numberofpkgs
, 'PP' billing
, 'CP' pkgstype
, 'ST' service
, '1' weight
, null Shippernr
from ExactOnlineREST..GoodsDeliveries del
join ExactOnlineREST..Accounts acc
on ACC.ID = del.DeliveryAccount
join ExactOnlineREST..Addresses ADR
on ADR.ID = DEL.DeliveryAddress
join ExactOnlineREST..Contacts CON
on CON.ID = DEL.DeliveryContact
where DeliveryDate between $P{P_SHIPDATE_FROM} and $P{P_SHIPDATE_TO}
order
by DEL.DeliveryNumber
It takes many minutes to run. The number of deliveries and accounts grows with several hundreds each day. Addresses and contacts are mostly 1:1 with accounts. How can this query be optimized for speed in Invantive Control for Excel?
Probably this query is run at most once every day, since the deliverydate does not contain time. Therefore, the number of rows selected from ExactOnlineREST..GoodsDeliveries is several hundreds. Based upon the statistics given, the number of accounts, deliveryaddresses and contacts is also approximately several hundreds.
Normally, such a query would be optimized by a solution such as Exact Online query with joins runs more than 15 minutes, but that solution will not work here: the third value of a join_set(soe, orderid, 100) is the maximum number of rows on the left-hand side to be used with index joins. At this moment, the maximum number on the left-hand side is something like 125, based upon constraints on the URL length for OData requests to Exact Online. Please remember the actual OData query is a GET using an URL, not a POST with unlimited size for the filter.
The alternatives are:
Split volume
Data Cache
Data Replicator
Have SQL engine or Exact Online adapted :-)
Split Volume
In a separate query select the eligible GoodsDeliveries and put them in an in-memory or database table using for instance:
create or replace table gdy#inmemorystorage as select ... from ...
Then create a temporary table per 100 or similar rows such as:
create or replace table gdysubpartition1#inmemorystorage as select ... from ... where rowidx$ between 0 and 99
... etc for 100, 200, 300, 400, 500
And then run the query several times, each time with a different gdysubpartition1..gdysubpartition5 instead of the original from ExactOnlineREST..GoodsDeliveries.
Of course, you can also avoid the use of intermediate tables by using an inline view like:
from (select * from goodsdeliveries where date... limit 100)
or alike.
Data Cache
When you run the query multiple times per day (unlikely, but I don't know), you might want to cache the Accounts in a relational database and update it every day.
You can also use a 'local memorize results clipboard andlocal save results clipboard to to save the last results to a file manually and later restore them usinglocal load results clipboard from ...andlocal insert results clipboard in table . And maybe theninsert into from exactonlinerest..accounts where datecreated > trunc(sysdate)`.
Data Replicator
With Data Replicator enabled, you can have replicas created and maintained automatically within an on-premise or cloud relational database for Exact Online API entities. For low latency, you will need to enable the Exact webhooks.
Have SQL Engine or Exact adapted
You can also register a request to have the SQL engine to allow higher number in the join_set hint, which would require addressing the EOL APIs in another way. Or register a request at Exact to also allow POST requests to the API with the filter in the body.
The SQL engine hides away all nifty details on what API calls are being done. However, some cloud solutions have pricing per API call.
For instance:
select *
from transactionlines
retrieves all Exact Online transaction lines of the current company, but:
select *
from transactionlines
where financialyear = 2016
filters it effectively on REST API of Exact Online to just that year, reducing data volume. And:
select *
from gltransactionlines
where year_attr = 2016
retrieves all data since the where-clause is not forwarded to this XML API of Exact.
Of course I can attach fiddler or wireshark and try to analyze the data volume, but is there an easier way to analyze the data volume of API calls with Invantive SQL?
First of all, all calls handled by Invantive SQL are logged in the Invantive Cloud together with:
the time
data volume in both directions
duration
to enable consistent API use monitoring across all supported cloud platforms. The actual data is not logged and travels directly.
You can query the same numbers from within your session, for instance:
select * from exactonlinerest..projects where code like 'A%'
retrieves all projects with a code starting with 'A'. And then:
select * from sessionios#datadictionary
shows you the API calls made:
You can also put a query like to following at the end of your session before logging off:
select main_url
, sum(bytes_received) bytes_received
, sum(duration_ms) duration_ms
from ( select regexp_replace(url, '\?.*', '') main_url
, bytes_received
, duration_ms
from sessionios#datadictionary
)
group
by main_url
with a result such as:
I have created a database with the intention of allowing analysis of patients blood test results for a certain condition. The query itself is working fine however is unsearchable (in other queries i've generated I've identified this is was because I had spaces in the headings and so removed these)
This query is now duplicating patients for each positive test they have from the selection. This seems to be linked to the .value column as goes away when this is removed however then the query becomes unsearchable using the quick search tool.
Is there any solution to this? I'm using the query wizard to build these. Believe the problem is with the 'OtherImmunology.Value' part
SELECT cci.rq6number,
pi.firstname,
pi.lastname,
pi.nhsnumber,
pi.dob,
Datediff("yyyy", [dob], [initialdiagnosisdate]) AS AgeAtDiagnosis,
pi.gender,
pi.[diagnosis(iapconsensuscriteria)],
cci.initialdiagnosisdate,
cci.steroidtherapy,
cci.responsetosteroids,
cci.igg4level,
cci.[igg4actualresult(mg/dl)],
cci.serumigg,
cci.[serumigg result(g/l)],
cci.otherimmunologytested,
cci.otherimmunology,
cci.** otherimmunology.VALUE,
cci **.[ca19-9level],
cci.[ca19-9actualresult(ku/l)],
cci.surgery,
cci.otherautoimmuneconditions,
cci.autoimmunecondition,
cci.completed
FROM [patient information] As pi
INNER JOIN [Clinical Contact Informtion] As cci
ON pi.[rq6number] = cci.[rq6number];
I found the following in the "e-mail" field of my newsletter subscriber database: ' OR 1=1/*
I know it's a SQL injection, but that's it. I've googled it a little bit, but I'm still on clear on what exactly it's trying to achieve. This occurred early Nov, and to my knowledge we had no outages around that time. Can any of you kind souls tell me what this guy was probably trying and do? Is there any way to know whether he achieved what he was trying to do?
I know virtually nothing about this and I'm worried. :(
'OR 1=1 is an attempt to make a query succeed no matter what
The /* is an attempt to start a multiline comment so the rest of the query is ignored.
An example would be
SELECT userid
FROM users
WHERE username = ''OR 1=1/*'
AND password = ''
AND domain = ''
As you can see if you were to populate the username field without escaping the ' no matter what credentials the user passes in the query would return all userids in the system likely granting access to the attacker (possibly admin access if admin is your first user). You will also notice the remainder of the query would be commented out because of the /* including the real '.
The fact that you can see the value in your database means that it was escaped and that particular attack did not succeed. However, you should investigate if any other attempts were made.
It probably aimed to select all the informations in your table. If you use this kind of query (for example in PHP) :
mysql_query("SELECT * FROM newsletter WHERE email = '$email'");
The email ' OR 1=1/* will give this kind of query :
mysql_query("SELECT * FROM newsletter WHERE email = '' OR 1=1/*");
So it selects all the rows (because 1=1 is always true and the rest of the query is 'commented'). But it was not successful
if strings used in your queries are escaped
if you don't display all the queries results on a page...
The specific value in your database isn't what you should be focusing on. This is likely the result of an attacker fuzzing your system to see if it is vulnerable to a set of standard attacks, instead of a targeted attack exploiting a known vulnerability.
You should instead focus on ensuring that your application is secure against these types of attacks; OWASP is a good resource for this.
If you're using parameterized queries to access the database, then you're secure against Sql injection, unless you're using dynamic Sql in the backend as well.
If you're not doing this, you're vulnerable and you should resolve this immediately.
Also, you should consider performing some sort of validation of e-mail addresses.
Its better if you use validation code to the users input for making it restricted to use symbols and part of code in your input form. If you embeed php in html code your php code have to become on the top to make sure that it is not ignored as comment if a hacker edit the page and add /* in your html code