Tag: Performance Probleme

SQL Server query optimization for Dynamics ERP

Posted on | by InfoMaster

SQL Server query optimization for Dynamics ERP

SQL Server query optimization for Dynamics ERP

1. Why perform SQL server query optimization for Dynamics ERP systems like Business Central or NAV?

In recent years, you've seen incredible improvements in many areas of database servers:

  • Database optimizers are constantly improving and finding ways to make queries more adaptable and overcome areas of poor optimization.
  • Memory speeds have increased massively due to advances in both memory technology and network bandwidth, CPUs have become much faster, and memory prices have dropped dramatically.

However, there are still professionals (like us) who make their living tuning queries and training others to optimize queries. This process involves finding certain slow queries (long durations) that are critical to Dynamics Business Central or NAV performance. These "slow SQL queries" can result from making strategic changes, whether in code, database structure, NAV server instance configuration, or something else. Optimizations are often necessary to ensure that these specific SQL queries execute consistently at a certain speed or meet a required performance standard.
But so why SQL server query optimization despite all the improvements?

Very simple:

  • Data volumes are also increasing dramatically
  • Customer/user expectations for performance/speed of applications like Dynamics Business Central have also increased
  • Customers/users expect not to wait for results now - they want to see current status immediately

2. Who should perform the SQL query optimization?

SQL query optimization is performed by:

  • Database Administrators
  • C/AL or AL developers who specialize in performance or specifically in databases.

Full-stack developers for AL and C/AL generally do not perform query optimization unless they have a specific interest or work experience. This is more of a specialization than a "quick learn" task, so most full-stack developers simply don't have time and need to hire a more specialized person to help them. Teams that don't have a readily available specialist can periodically bring in consultants like us the DynamicsProject Team to help.

There are also many database administrators who manage SQL databases where only "basic" availability and performance are required. These SQL databases are used by cost-conscious organizations that don't need to tune every database like a "race car": Most of their SQL databases are used by internal users who are used to Dynamics ERP performance and speed moderation.

3. What skills are required in SQL query optimization?

Most of us are not wizards at TSQL, but many DBAs have gotten pretty good at query tuning (e.g., through our V8 Performance Workshops).

A very unscientific estimate of what skills make someone in query optimization:

40% - The time, interest, and resources (including a network of people to ask) to build a comprehensive profile of "performance information" - this is patterns that don't optimize well, boundary conditions where performance becomes poor, understanding trace flags, and configuration on TSQL.

30% - Interest and ability to learn how the database engine optimizes and processes queries using indexes and other resources, and what affects parallelism when running multiple queries simultaneously against a live database.

10% - Understanding of the TSQL language and various ways to rewrite a query in conjunction with Dynamics AL or C/AL programming to produce a specific result set. This often only works with database DevOps.

What is database DevOps?
Let's start with the definition of DevOps. DevOps is a set of practices that combine software development (dev!) and IT operations (ops!) with the goal of delivering more features, fixes, and updates faster, in alignment with business goals. Database DevOps applies the same principles and ensures that AL or C/AL database code is included in the same process as development code.

Database DevOps helps teams further identify and streamline the application development and release process by addressing a known bottleneck: AL or C/AL Dynamics source code changes.

4. SQL query optimization tools and their cooperation.
4.1 Execution plans

How the SQL query from Dynamics Business Central is executed behind the scenes.

  • "Estimated" execution plans show the decisions the optimizer has made to execute the query, including the estimated number of rows that will flow through the different parts of the plan
  • "Actual" execution plans are estimated plans that are updated with runtime statistics, e.g. how many lines have passed through the plan. If the plan is "adaptive", it contains some information about which options were chosen
4.2 Query memory

SQL Server 2016+, all editions

  • This function tracks execution plans and aggregated runtime metrics (duration, CPU usage) along with aggregated wait statistics
  • This also has the possibility to "freeze" plans
  • Query store information is restored with the database itself so that it can be shared between environments as needed.
4.3 Dynamic Management Views and Performance Indicators (DMV)
  • These help to understand the bottlenecks of the whole instance during slow performance
  • Example: Overall maintenance statistics for the instance and metrics on memory latency during the time the queries were poorly executed can help explain whether the query really needs to be optimized
  • Query optimization often requires callbacks for "workload optimization"
  • SQL trace and enhanced event traces (enhanced events are a lightweight performance monitoring system to collect data and are the foundation of V8 Search XE).
  • The "Old" SQL Profiler is difficult to use for query optimization in "live" operations, as they can easily slow down your workload and cause performance issues when tracing.
  • Execution plans (filtering does not help in this case, the plans are all examined / collected and the filter is applied too late).
  • Wait Statistics (filtering can help here, but the collected data is so massive that you have to be very careful - and sorting and querying the collected data is also quite cumbersome
  • "Business Central Server Trace Events" There are two event trace providers that publish different trace events in the event log: Microsoft-DynamicsNAV-Server and Microsoft-DynamicsNAV-Common. The Microsoft-DynamicsNAV-Common provider is for telemetry trace events only. All other events use Microsoft-Dynamics NAV server. Usually, you need to specify the event trace provider in the monitoring tool you use (e.g. V8 Search XE)
  • "SQL Trace Events" trace a specific set of SQL statements executed by the Business Central Server instance against the Business Central database on SQL Server

5. Difficult problems - dispute over resources

It's hard to predict how SQL queries will interact with each other in a live workload

Shared resources:
  • Memory for queries - a certain amount of memory must be allocated for sorting/linking/moving data in a query. Many queries running at the same time that require a lot of memory can cause problems with this. (Sometimes queries assume they need much more of this memory than they need, and it needs to be optimized - this will probably not be aware to users outside of a live workload).
  • The number of queries that make changes and the approach to blocking are difficult to predict outside of a live workload (changes in query plans can cause blocking when they were not previously present)
Changes in server resources - even improvements - can cause blockages if they were not there before
  • Example: Switching to a new server with more memory and faster CPUs resulted in an increase in lock wait times due to reduced in-memory wait times and faster query execution.

6. Is a check in the "live" SQL database required?

Yes. One example of this is parallelism.

Optimizing the degree of parallelism for a workload and for specific queries in that workload is usually quite hardware specific, and you need a live environment.

Workload- „Replays“ sind im Toolkit von SQL-Servern verfügbar, aber sie sind:

  • Repetitions only - you can't meaningfully "boost" the activity (deleting the same lines 10 times is not the same as deleting different lines 10 times)
  • Time-consuming to set up

7. Automated SQL server query optimization: history and development
7.1 Automated plan correction

SQL Server 2017+, Enterprise Edition

  • Built on the query store
  • Detects SQL queries that are sometimes fast and sometimes slow
  • Can only recommend changes if desired and set up
  • Can freeze plans, test to see if it helps, and respond accordingly (Freezing is intended as a temporary solution - it is recommended that a user evaluate the query for optimization as a longer-term solution)
  • Very good function for identifying parameter sniffing
7.2 Intelligent Query Processing
  • "Intelligent Query Processing" (IQP) have been released in the latest versions of SQL Server
  • Several of these features fix common query optimization issues in SQL Server
  • More information: Intelligent Query Processing in SQL Databases.

8. Common errors and pitfalls in SQL Server query optimization.
  • Lack of connection between DBAs and Dynamics development teams
  • Lack of knowledge of the execution plans in the team
  • Lack of understanding of SQL Server isolation levels and "optimistic" options
  • Lack of knowledge of how Dynamics AL or C/AL programming is implemented on the SQL Server

We will be happy to personally answer any further questions you may have on this topic.
Your dynamicsproject.com Team

Dynamics 365 Business Central Wave 2

Posted on | by InfoMaster

Dynamics 365 Business Central 2020 Release Wave 2 is the first release that does not include the classic development environment (also referred to as C/SIDE). With 2020 Release Wave 2, users access Dynamics 365 Business Central via the browser, the Windows 10 desktop app, mobile apps on Android and iOS, or in Outlook.

Dynamics 365 Business Central 2020 Release Wave 2

Microsoft is accelerating its investment in speed and productivity features for the modern browser experience, reaching a major milestone in transforming it into a world-class desktop experience for new and experienced Dynamics users. The modern clients now support so many productivity features that the older Dynamics NAV Windows client for Dynamics 365 Business Central will not be available in the future. The older Dynamics NAV client will continue to be supported in earlier versions according to the support lifecycle.

Strengthening of independent software vendors (ISV)

Meanwhile, the 2019 Release Wave 2 update offers a number of features designed to simplify independent software vendor (ISV) development for new solutions and, in particular, streamline migration from the source code customization model of Dynamics NAV to Dynamics 365 Business Central. In addition, Microsoft will focus on streamlining the path for independent software vendors (ISVs) to bring their solutions - and therefore their customers - online to Dynamics 365 Business Central.

Modern developer tools

The modern development environment based on Visual Studio Code with Azure DevOps and an AL language now supports large app development. Therefore, C/SIDE for Dynamics 365 Business Central will be discontinued in the future. Therefore, we have adjusted ourselves and our performance analysis tool "V8 Search XE" to the new situation. Nevertheless, there will be some changes in the area of performance optimization of Dynamics 365 BC Wave 2 and SQL Server.

 How to work with performance issue in Dynamics 365 Business Central 2020 Release Wave 2 in the future....

What do you do when users complain that "it's slow" in the new modern browser environment? In this section, we would like to describe a troubleshooting process that can help you find the main cause of the problem.
Before you start solving a performance tuning problem, it is often helpful to define and quantify "slow". Try to negotiate acceptable values for the execution time of "slow" operations with BC users. This is sometimes referred to as "establishing a baseline".

To solve a performance problem, the following iterations are often performed:

  • Measure system performance and capture performance data from SQL Server and Dynamics 365 BC servers
  • Search for a bottleneck
  • Remove the bottleneck

and continue until the "slow" operations are comparable to the established baseline.

Monitoring and analysis of telemetry Business Central 2020 Release Wave 2 and higher.

Business Central sends telemetry data for various activities and operations. V8 Search XE collects the telemetry data for Business Central On-Premises via the V8 Services. This means the data is stored centrally for all servers in the V8 database.
By monitoring the telemetry, you get a overview of the activities and overall health of your Dynamics system. It helps you diagnose problems and analyze processes that affect performance.

The following article may help you find more on this topic:
Monitoring and Analyzing Telemetry Business Central | Microsoft Docs

Conclusion

In our opinion, Microsoft has provided many new features in the area of performance optimization for Dynamics Business Central 2020 Release Wave 2. On the other hand, system maintainers and administrators will also have to deal with AL programming just like C/AL developers. We agree with the widespread opinion that we may continue to have our fun with the Dynamics Business Central / NAV onPremise versions for some time with the C/AL programming language. Of course, the know-how about the older NAV / BC versions will continue to play a crucial role for an ERP developer and administrators in the future.
Things change but sometimes also very quickly.

We will be happy to personally answer any further questions you may have on this topic.
Your dynamicsproject.com Team

Optimize SQL Server performance with Dynamics 365 Business Central on-prem

Posted on | by InfoMaster

Optimize SQL Server performance with Dynamics 365 Business Central on-prem and Dynamics NAV (all previous versions). There are so many factors that contribute to performance issues in the SQL world that it can be downright confusing where to even start.

As the Dynamics application and associated SQL databases grow, things change. Rows are changed, schema is updated, and often things slow down. These performance issues can occur suddenly or intermittently, but it's important to know how to distinguish performance issues so you can fix the individual problem.

The post looks at some steps you should take if your SQL database is behaving strangely. So you can diagnose exactly what is going wrong and focus on speeding things up. Optimizing SQL Server Performance with Dynamics 365 Business Central should start with checking your SQL Server configuration, if possible.


Is it the database? Or the server?

If you have access to the server, you can use a tool like System Monitor or Perfmon to check the processes running on the server itself. High memory usage, high CPU usage, and slow network traffic are red flags where the problem may not be the database itself.

There are many other system diagnostic tools e.g. V8 Search XE that can help with this process. However, one of them should at least point you in the right direction before you spend countless hours looking at tables, indexes and more.

SQL Server has an excellent profiling tool that can both fix existing problems and isolate slow-running queries. XEvent Profiler displays a live view of extended events. It's also a fantastic way to retrieve additional data about your database for performance optimization purposes (e.g. see which queries are executed most often, examine things like memory and CPU usage, identify potential blocks, etc.).

In V8 Search XE, optimizing SQL Server performance with Dynamics 365 Business Central is divided into several areas.


SQL Server System Check
V8 Search XE SQL Optimierung
V8 Search XE SQL Optimierung

V8 Search XE is infinitely useful and a good first step in diagnosing poor performance (or errors in general). It is useful in both development and production environments.

By default, all databases in the SQL Server instance are eligible for the various database-specific checks, and you can use the optional parameter to restrict these checks to specific databases. Valid from SQL Server 2012 onwards.


Performs checks:
  • Processor
  • Memory
  • Pagefile
  • I/O
  • Server
  • Service Accounts
  • Instance
  • Database and tempDB
  • Performance
  • Indexes and Statistics
  • Naming Convention
  • Security
  • Maintenance and Monitoring

These various performance-related queries may be worth reviewing periodically to find potential problems that may arise over time.


Found the perpetrator? Find out now why the query is slow.

Once you have identified a particular query that has been identified as running slowly in the Analysis views in the V8 Search XE "SQL Server Check", you can view the execution plan to see exactly what SQL Server is doing behind the scenes. You can access it from a specific query simply by, the "Show Plan" button in the toolbar or from the ribbon:

V8 Query Plan

Execution plans can easily reveal potential indexing issues that cause your entire table to be scanned instead of looking for exactly what it needs. At first glance, they may seem incredibly complex (and they can be), but once you've worked with them, you'll learn to identify patterns and what they're associated with (e.g., the X operation indicates a missing index for a table, etc.).


A few things to look for here:
  • Look for Warnings - Warnings like "No Join Predicate" should be very obvious red flags that you probably need to address. Most warnings should at least warrant further investigation.
  • Order of operations (cost) - Consider the most expensive operations and determine whether such ordering makes sense. Does a simple join consume 90% of the processing power for the entire call? If so, something could be wrong.
  • Scans vs. Seeks - Neither is necessarily bad, but if one takes much longer than expected (one of them), it's probably worth determining if an index is missing (i.e., SQL Server scans the entire table instead of grabbing a well-defined seek value).

Understanding execution plans comes with experience, and hopefully you won't have to deal with them too often. But if you do, know that they can be a valuable ally in the fight against poor performance.


Investigate possible bad / missing indexes

SQL indexes are the A and O for performance optimization in your Dynamics NAV SQL database. You can think of them as similar to the indexes you might find in a book, in that they allow SQL to "know" where to look for specific data, rather than arbitrarily flipping through page after page.

It's worth noting that indexes are not "error-free" and as with most things, if you use them incorrectly, they can do more harm than good (i.e. imagine going on a scavenger hunt with exceptionally trails or lousy clues).

One great thing is that you don't always have to do this work yourself. People like us, have developed tools like V8 Search XE to recommend indexes based on table usage. Invaluable for optimizing SQL Server performance with Dynamics 365 Business Central on-prem and Dynamics NAV!


Statistics

As valuable as it can be to optimize SQL queries, it's worth noting that SQL Server does some of these "well" and others "poorly"by itself. This is accomplished by using statistics to monitor calls made, caching execution plans, and making judgments about how a particular call could/should best be executed.
Note that we mentioned "good or bad" and that is intentional. Statistics, while incredibly valuable, can also cause problems if they are not captured correctly or are outdated. In most scenarios, you probably won't need to deal with them on a regular basis, but it's important to know that they exist.


As with almost any technology, it is very unlikely that just one of these items will cure all of your performance issues with SQL Servers and Dynamics 365 Business Central or Dynamics NAV. Consider all of these as valuable tools in your troubleshooting arsenal and use them together when problems arise.


We will be happy to personally answer any further questions you may have on this topic.
Your dynamicsproject.com Team

Performance Probleme in Dynamics NAV

Posted on | by InfoMaster

Performance Probleme in Dynamics NAV

 

Performance Probleme in Dynamics NAV ist auch unter NAV 2017 und dem SQL Server 2016 immer noch ein Thema.

Vor einigen Tagen habe ich eine neue virtuelle Maschine mit Dynamics NAV und dem SQL Server 2016 aufgesetzt, um ein Performance Troubleshooting mit dem „Query Store“ (Abfragespeicher) zu testen.

Ich bin ein leidenschaftlicher SQL Server Datenbankarchitekt, Software Engineer und ein Dynamics NAV Performance Troubleshooter. Kann ich mich in Zukunft nach einem anderen Betätigungsfeld umsehen, da ihnen nun der SQL Server 2016 mit dem sogenannten Abfragespeicher („Query Store“) ein Troubleshooting Tool in die Hand liefert, mit dem sie selbstständig Ausführungspläne „Plan Regressions“ analysieren und schlussendlich beheben können. Werden auch die teilweise hochpreisigen Performanceüberwachungstools von Drittherstellern, wenn sie es nicht schon sind, überflüssig?

Ich fände es gut!

Klar, unser Team lebt teilweise von den Problemen anderer mit Dynamics NAV und dem SQL Server. Sehr oft werde ich bei verschiedenen Dynamics NAV und SQL Server Blocking Problemen kontaktiert, um diese mit den „Vision 8 Tools“ effizient zu lösen. Sehe ich den „neuen“ Kollegen als Gefahr oder als Bereicherung für die Dynamics NAV Welt an?

Ich möchte ihnen das neue SQL Server Tool kurz vorstellen und ein wenig zeigen, wie gut bzw. schlecht es seinen Job macht.


Um was handelt es sich dabei bei dem neuen Feature „SQL Server Query Store“?


Mit dem neuen Feature SQL Server Query Store lassen sich Informationen zur Ausführung von Abfragen auf z. B. Dynamics NAV Datenbank aufzeichnen. Somit sind z. B. auch nach einem Neustart des SQL Server Dienstes die verwendeten Ausführungspläne noch verfügbar. Die Ausführungspläne können zusammen mit Statistiken (wichtig für Performance Optimierung) zur Ausführung der Abfrage über vorgefertigte Reports komfortabel analysiert werden.





Ich sehe den „SQL Server Query Store“ als eine großartige Ressource und Ergänzung für Performance Tuning. Allerdings sollten sie die neue Möglichkeit der Performance Optimierung sorgfältig verwenden!


Was können sie tun, wenn immer wieder Performance Probleme in Dynamics NAV auftauchen?


Die meisten meiner Kunden haben komplexe Prozesse in ihrem Dynamics NAV abgebildet, die die Leistung beeinträchtigen, wobei das größte Problem dann Sperren, Deadlocks und Timeouts sind.

Während die Dimensionierung des NAV-Servers gut eingerichtet ist, sind im Bereich der SQL Server-Dimensionierung, Einrichtung, proaktive Überwachung, vorbeugende Wartung und Optimierung sehr oft nicht vorhanden. Die „SQL Server HW und SW Plattform“ ist ein sehr wichtiger Aspekt, wenn es um die richtige Leistung von NAV geht. Die Lösung könnte so einfach sein wie das Hinzufügen von RAM oder das Ändern von Festplatten.

Allerdings sobald alle diese Schritte durchgeführt wurden, sind Leistungsprobleme im stark individualisierten Dynamics NAV Lösungen immer noch wahrscheinlich.


In diesem Zusammenhang empfehle ich ihnen unsere „Vision 8 Performance Packs“.

Leider kommt man oft um ein Redesign für NAV (Pages, Berichte usw.), Datenarchivierung, Re-Codierung spezifischer Anwendungsbereiche nicht drum rum, um eine optimale NAV-Leistung zu erreichen.


Toller Hinweis…, aber wo fängt man und welche Tools helfen „Bad Code“ zu finden?


Aus meiner Sicht werden viele wichtige Tools im Bereich Performance Optimierung ab dem SQL Server 2012 mit liefert z. B. DTA (Database Tuning Advisor), Extended Events, Hypothetical Indexes, Performance Counters, SQLDiag (Sammeln von Diagnoseinformationen), SQLIO (Disk Subsystem Benchmark Tool) u.v.m.

Ab dem SQL Server 2014 SP 2 gibt es den neuen Kollegen „Query Store“, DBCC CLONEDATABASE (Kopie einer Datenbank nur mit Schema und Statistiken), In-Memory OLTP (Arbeitsspeicheroptimierung) usw.

Aber auch Dynamics NAV bietet eine Menge Tools und Funktion um Performance zu messen und zu optimieren z. B. „FULL SQL Trace“, „Data Collector Set“ für Microsoft Dynamics NAV „trace event data“ oder Dynamics NAV „Performance Counters“.

Leider können diese Tools nur einen Anhaltspunkt auf suboptimalen C/AL Code in Dynamics NAV liefern.


Aber im welchem Dynamics NAV Objekt ist der „bad“ C/AL Code?


Mit dieser Frage, beschäftige ich mich seit dem Jahr 2012 mit dem erscheinen von Dynamics NAV 2013. Aus diesem Grund habe ich von 2013 bis 2016 das „V8 NAV SQL Studio“ entwickelt. Für mein Team und mich ist bis heute das V8 Studio im Bereich Dynamics NAV, das „NAV Tool“, wenn es um C/AL Code und NAV Objekte geht. Von 2014 bis Sommer 2016 hatten wir das V8 Studio der „Dynamics NAV Welt“ angeboten – allerdings mit mäßigem Erfolg!



Woran lag es, dass NAV Entwickler, Administratoren, Dynamics-NAV-Berater oder Projekt Manager nicht damit arbeiten wollten?

Es ist zu komplex!

Ein guter Kollege sagte zu mir, dass er oft auch unter Zeitdruck im Daily Business steht und keine Möglichkeit hat sich mit diesem Thema zu beschäftigen.

Okay. Also habe ich das Tool vom Markt genommen und aus Teilen des V8 Studios ein neues Tool entwickelt – den „V8 NAV SQL Monitor“.

Ich bin mir sicher, mit der richtigen Beratung (natürlich durch mich) und dem Einsatz der richtigen Kombination der vorhandenen Tools, finden sie ihren „bad“ C/AL Code und minimieren Performance Probleme in Dynamics NAV!


We will be happy to personally answer any further questions you may have on this topic. Simply contact us via our contact form or by e-mail to info@dynamicsproject.com!

 

Olav Treffurt