Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 5 years ago.
Improve this question
I have a broad question I would like some advice on.
I'm a developer on a 3-person scrum team working on the same application (c# web form application). For years, the process has been that developers run a local IIS site on their machine and point their application at a shared database. Using the shared database has many benefits over local SQL instances for our business and minimizes the overhead for each developer on setup and maintenance.
This process has worked for years (mainly because everyone was in the office), but now half of us work from home or remotely. SQL performance through the VPN is atrocious, and ends up slowing our local sites down at an order of magnitude when processing the SQL calls through the VPN. Because of the slowness, it's really caused slowed down my productivity and I'm looking for options.
I've thought of the following solutions:
Run SQL locally
Setup Replication
Try and point a SQL instance at a UNC path for mdl files (if this is possible)
Tweak VPN settings to speed up MS-SQL calls
I'm doubtful there is a way to speed up the traffic going through the VPN, but I'd be open to any ideas.
RDP into my machine at work when I work from home. I truly hate doing this.
Below are the technical specs of our setup:
MS-SQL 2008 R2
C# Web Forms
N-Tier Architecture
Have any other development teams had this issue before, if so what where some of the solutions?
Thanks
Regardless of the technology you are using, this situation happens to many developers who work remotely.
Your three options are viable but consider this:
Running a local database server introduce the possibility of environmental errors. If you have 3 developers, you now have 3 different databases. Even if the structure of the DB itself never changes, the test data will certainly differ from one developer to the other. At some point you may have to reconcile the "master" version of the test data with each local instances.
Tweaking the VPN settings is certainly worth looking into but I cannot help you with that. I have no idea how you would do this.
RDP into your remote machine could be the best option. This will guarentee you have the same setup when you work from home or from the office. I worked in two different large companies before and that's how we were doing it in both places. May I ask why you hate this? It's true sometimes it doesn't work or it is very slow.
You may end up doing all these options. I mean tweaking your network speed is always good, no matter how your work. You could have local instances on which you do your development then you synchronize your work with the repository and RDP to your office machine to test your new code with the master instance of the database. This way you don't have to work on RDP all day and you end-up testing your code twice, in two different environments. That's good, right?
Related
Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 5 days ago.
Improve this question
I have a database (MySQL, AWS RDS). This is a production database that has customer information including names, emails, bank account information. Some of it is encrypted, some of it is not.
We want to setup an environment that can be regularly used for automated testing. We want the database for the test environment to be the same as production except we want to replace customer data.
We want to do this in a way where the customer data never leaves the production environment. We don’t mind creating an “intermediate” environment that may initially contain some customer data but then gets removed. From the intermediate environment, we’d transfer the cleaned database to the testing environment.
Appreciate the guidance since I’m way out of my depth here
There is no easy / automated solution to do this that I am aware of. You need to replicate your data to a different system and have that replication service scrub your data for you. A few options come to mind:
You could write a batch processor that dumps the DB down to disk, loads it into a secondary server (staging/scrubbing environment), and then run a series of cleanup scripts. Then you can again dump the data down.
You could write a database trigger that fires on the events you care about and maintains a staging table of sanitized data.
You could write a test data generator that users the patterns of your production data to generate fake data in a testing table. There are many tools that can help with this, both open source and commercial.
Personally, I lean towards the last option because it's the safest and can be used in many places, like on a local dev machine, CI/CD system, shared staging environment, etc. I do believe there is a strong case for sending a copy of a subset of production data to canary systems as part of a rollout strategy, though. Effectively testing your release with live data before wiring it to your live database.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I work in a company on a huge application made with Delphi 10 Seattle (VCL), using direct connection to Oracle database server (2-tier).
One of the experts proposed to migrate to the 3-tier architecture using mormot (or another library/components).
One of his arguments that the new architecture (3-tier) will provide more performance (time of exchange data), because https is faster than direct oracle connection and json objects are much lighter (without using a cache policy on the rest) and we can after that make web clients.
I didn't understand:
Why is https faster than oracle connection protocol? (if it's true, why oracle doesn't use https and json as protocol for exchange data?).
Isn't a security problem if we let all the functions and queries on the client side (even if we will do web clients)?
Cordially
In the context of any n-Tier framework, remote SOA access via a REST solution could have performance benefits.
The mORMot documentation tries to introduce all those concepts (SOA, REST, MVC, n-Tier, Clean Architecture, DDD...) and, then only, speaks about performance impact. Don't take a documentation sentence in isolation. Consider the whole picture, and the use-cases.
Why is https faster than oracle connection protocol?
HTTPS is not "faster" in se. What was meant is that it could be more efficient for a remote connection, especially over the Internet.
The Oracle Connection protocol was designed to run on a local network, whereas HTTP is a query/answer model.
The main PRO of the Oracle protocol is that it is more complete (complex?) than a simple query/answer model: it can dialogue with the Oracle server to cache statements and data, it can have real-time notifications, it can prepare the data in binary form ready to be encoded.
In terms of performance, the main CON of the Oracle protocol is that it requires more round-trips other the wire: it was designed to work on local network, with a low latency. Over an Internet connection, it will be much slower, and, for security reasons, is very likely to be encapsulated into a VPN - reducing even more the performance.
We should not speak of "performance" in an abstract way. There are several ways to skin a cat... If you want raw query performance, use another kind of database, like Redis.
But for business applications, the main "performance" point is perhaps more about scaling. And here, the Oracle protocol has a bigger cost in its database connections. Maintaining a connection, especially maintaining transactions, can be very demanding. You can maintain up to a few dozen/hundredths of simultaneous DB connections on a typical Oracle server. Whereas it is very easy to have a REST server maintaining thousands of simultaneous clients. And even if you currently expect only a few clients, how could you imagine your future? All serious applications expect a REST-like interface, nowadays. And keep the database connection local on the server side.
Isn't a security problem if we let all the functions and queries on the client side (even if we will do web clients)?
Security is another concern, and here a REST Web client has known and proven strategy, with proper audit methodology. You will never "let all functions on the client", in a REST interface. The framework offers several ways of authentication and authorization - check the documentation, from URI-signature to JWT.
Opening an Oracle endpoint to the whole Internet is not a good idea, in terms of security and scalability. Even Oracle is offering dedicated solutions for proper networking.
Anyway, a framework like mORMot was designed to offer REST, SOA, ORM and web MVC in a single package, and performance was driven from the ground-up - as a bonus. If you expect to design a RAD VCL/FMX application, go with direct database connection, and be data-centric. If you want something more open and maintainable, consider SOA, and be service-centric. Today, I develop all my SOA solutions as Micro-Services, with stand-alone databases, and mORMot as tooling, with huge performance - up to million of data items per second.
Sounds like a vague story to me, and I'd have a hard time believing this unless I got well documented proof. And even then, it's easy to compare apples and oranges when comparing the performance of such different ways of architecture.
I don't think that https in general is faster than a direct connection, but it depends on a lot of variables.
Moreover, mORMot itself needs to connect to the database as well, for which it can use some Direct Oracle Access (I assume the same as the one you compare it with), or OleDB (aka ADO) which is in general the same or slower than DOA, so there is no gain there. There is only the extra overhead of mORMot. See software architecture of mORMot.
So, how can it be better?
If the middle tier uses connection pooling when the client cannot, for instance. In that case, having a middle tier for pooling connections can lead to better performance, because no new connection needs to be established on every request. This can save a lot of overhead.
The same goes for caching. If you want to build a web site or web app, having a middle tier can greatly improve caching, if query results can be cached regardless of user. If you have client side caching, you cache it for that user/browser/session only, while in the middle tier, you can cache some data for all your visitors, which is a great benefit.
If there is a lot of processing needed before the database is involved. This can be faster on a high end middle tier. Then again, if you have lots of users, you might run out of hardware capacity (or cloud budget), and you might consider doing part of the legwork on the client.
So there are all kinds of benefits to 3-tier. whosrdaddy already named scalability, portability and redundance. Performance can be one of the benefits as well, if you tick any point like the ones listed above, but in general it's not a main reason for going from 2-tier to N-tier.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
We currently use SQL CE databases on the client machines, which then synchronise their data to a central server using the merge replication/RDA functionality of MS SQL. The amounts of data involved is small, and the central server will often be idle for ~95% of the time - it's only active really when data is incoming, and is typically synchronised on a daily/weekly basis.
The SQL Standard licensing costs for this are large, relative to the SQL server workload / the amount of data we're talking about (in the order of 100s of MBs maximum). What I'd like to know is if there's an open source alternative (mySQL or similar) which we could use as the backend data storage for our .NET application. My background is Windows Server Admin, so relatively new to Linux, but happy to give it a go and learn some new skills, as long as it won't be prohibitively difficult. If there are any other alternatives that would be great too.
Well this is quite a open ended question so I am going to give you some guidelines around what you can start researching.
Client Side embeded databases.
MySQL can be embedded just from my understanding MySQL as a embedded server might be overkill for a client.There are however a stack of alternatives. Once such a point would be the Berkely database system. There are other alternatives as well. Keep in mind you dont want a FULL sql server on the client side you are looking for something light weight.You can read about Berkley here: http://en.wikipedia.org/wiki/Berkeley_DB and about alternatives here : Single-file, persistent, sorted key-value store for Java (alternative to Berkeley DB). They mention SQLite which might just be up your alley. So in short there is a whole stack of open source tools you can use here.
Back End Databases. MySQL will do the job very well and even PostgreSQL. PostegreSQL seems to support more enterprise features the last time I looked however that might have changed. These two are your main players in the SQL server market as far as open source is concerned. Either one will do fine in your scenario. Both PostgreSQL and MySQL run on windows as well so you dont have to install Linux though I would suggest that you invest the time in Linux as I have it is well worth the effort and the peace of mind you get is good.
There is one major sticking point for you if you switch over to MySQL/PostgreSQL that the current RDA/replication technology you have will not be supported by these databases and you will need to look at how to implement this probably from scratch. So while the backend and even front end DB's can be replaced the replication of the data will be a little more problematic but NOT impossible.
Go play with these technologies do some tests and then you will need to decide how you will replace that replication.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I would like to create a database with the goal of populating this database with comprehensive inventory information obtained via a shell script from each client machine. My shell script currently writes this information to a single csv file located on a server through an ssh connection. Of course, if this script were to be run on multiple machines at once it would likely cause issues as each client potentially would try to write to the csv at the same time.
In the beginning, the inventory was all I was after; however after more thought I began to ponder wether or not much much more could be possible after I gathered this information. If I were to have this information contained within a database I might be able to utilize the information to initialize other processes based on the information of a specific machine or group of "like" machines. It is important to note that I am already currently managing a multitude of processes by identifying specific machine information. However pulling that information from a database after matching a unique identifier (in my mind) could greatly improve the efficiency. Also allowing for more of server side approach cutting down on the majority of client side scripting. (Instead of gathering this information from the client machine on the startup of each client I would have it already in a central database allowing a server to utilize the information and kick off specific events)
I am completely foreign to SQL and am not certain if it is 100% necessary. Is it necessary? For now I have decided to download and install both PostgreSQL and MySQL on separate Macs for testing. I am also fairly new to stackoverflow and apologize upfront if this is an inappropriate question or style of question. Any help including a redirection would be appreciated greatly.
I do not expect a step by step answer by any means, rather am just hoping for a generic "proceed..." "this indeed can be done..." or "don't bother there is a much easier solution."
As I come from the PostgreSQL world, I highly recommend using it for it's strong enterprise-level features and high standard compliance.
I always prefer to have a database for each project that I'm doing for the following benefits:
Normalized data is easier to process and build reports on;
Performance of database queries will be much better due to the caching done by the DB engine, indexes on your data, optimized query paths;
You can greatly improve machine data processing by using SQL/MED, which allows querying external data sources from the database directly. You can have a look on the Multicorn project and examples they provide.
Should it be required to deliver any kinds of reports to your management, DB will be your friend, while doing this outside the DB will be overly complicated.
Shortly — go for the database!
I was originally planning on using a local machine on our network as the development server.
Then I had the idea of using a subdomain.
So if the site was at www.example.com then the development could be done at dev.example.com.
If I did this, I would know that the entire software stack was configured exactly the same for development and production. Also development could use the same database as production removing the hassle of syncing the data. I could even use the same media (images, videos, etc.)
I have never heard of anyone else doing this, and with all these pros I am wondering why not?
What are the cons to this approach?
Update
OK, so its seems the major no no of this approach is using the same DB for dev and production. If you take that out of the equation, is it still a terrible idea?
The obvious pro is what you mentioned: no need to duplicate files, databases, or even software stacks. The obvious con is slightly bigger: you're using the exact same files, databases, or even software stacks. Needless to say: if your development isn't working correctly (infinite loops, and whatnot), production will be pulled down right alongside with it. Obviously, there are possibilities to jail both environments within the OS, but in that case you're back to square one.
My suggestion: use a dedicated development machine, not the production server, for development. You want to split it for stability.
PS: Obviously, if the development environment missed a "WHERE id = ?", all information in the production database is removed. That sounds like a huge problem, doesn't it? :)
People do do this.
However, it is a bad idea to run development against a production database.
What happens if your dev code accidentally overwrites a field?
We use subdomains of the production domain for development as you suggest, but the thought of the dev code touching the prod database is a bit hair-raising.
In my experience, using the same database for production and development is nonsence. How would you change your data model without changing your code?
And also 2 more things:
Its wise to prepare all changes in SQL script, that is run after testing from different environment not your console. Some accidental updates to live system made me headake for weeks.
Once happend to me, that restored backup didn't reproduced live system problem, because of unordered query result. This strange baviour of backup later helped us find the real problem simplier, than retrying on live system.
Using the production machine for development takes away your capacity to experiment. Trying out new modules/configurations can be very risky in a live environment. If I mess up our dev machine with an error in the apache conf, I will just slightly inconvenience my fellow devs. You will be shutting down the live server while people are trying to give you their money.
Not only that but you will be sharing resources with the live enviroment. You can forget about stress testing when the dev server also has to deal with actual customers. Any mistakes that can cause problems on the development server (infinite loop taking up the entire CPU, running out of HDD space, etc) suddenly become a real issue.