On 29th September we will have SQL Saturday Porto and I’m proud to announce that I will be speaking!
I will be speaking about dbachecks PowerShell module – dbachecks – Checking SQL Server environments with PowerShell.
The event will be held at ISEP, Rua Dr. António Bernardino de Almeida, 431, 4249-015 Porto, Portugal and you can see the full schedule here.
Register, it’s free!
Hope to see you in Porto! If you see me, come and say hi!
Starting on 19th of July and during 3 days the Tuga IT 2018 Summer Edition will be happening in Lisbon!
This edition will have:
4 full-day workshops
38 breakout sessions (1 hour)
5 deep dive sessions (2 hours)
That is over 72 hours of content delivered by 44 awesome speakers, many of them recognized industry experts.
You can see the full schedule here. This edition will have sessions about: Cloud, Data Platform, Development, Integration, IOT, Office, PowerShell and Security.
I will be speaking on 19th about the dbachecks PowerShell module for SQL Server professionals. You can find more information about the session here: dbachecks – Checking SQL Server Environments With PowerShell.
The event will be held at Microsoft Headquarters in Lisbon.
What are you waiting for? Registration to the main event is free for all attendees, with the exception of the full-day workshops.
Main event registrations
Workshops registrations.
Hope to see you in Lisbon!
On one of the last clients I have worked, I have implemented dbachecks in order to get the state of art and know how cool or bad the environments are before start knocking down the bad practices.
This client has seven different environments with more than 100 instances and more than 2000 databases.
A non-parallel execution (single session) took more than 2 hours.
This is not a big problem when we run it out of hours and we don’t want/need to be looking or waiting for it to finish.
I set up it to run at 6 in the morning and when I arrive at the office I can refresh the Power BI dashboard a get a recent state of art.
Well, 2 hours seems not too much if we compare with some other dbachecks users
What if we want to lower down that 2 hours execution? Or the Hiram Fleitas’ (b | t) 4,6 hours?
First let me remember that this doesn’t come for free! I mean, if you set multiple checks in parallel (different sessions), you will consume more resources (CPU, memory, etc) on the machine where you are running them. Test the process and find a reasonable limit for it, otherwise this can become slower that the serial execution.
By default, dbachecks works with the values previously saved (for that we use Set-DbcConfig or the Import-DbcCheck). This means when we start a new session and the last session have changed any configuration, that configuration is the one that will be used in the new session.
Imagine that I want to check for databases in Full Recovery Model on the production environment and I want to start (in parallel) a new check for the development environment where I want to check for Simple Recovery Model if this setting is not changed in the correct time frame, we can end checking for Full Recovery Model on the development environment where we want the Simple Recovery Model.
The first time I tried to run tests for some environments in parallel, that had the need to change some configs, I didn’t realise about this detail so I ended up with much more failed tests than the expected! The bell rang when the majority of the failed tests were from a specific test…the one I had changed the value.
-Temporary parameter for the rescue!On my last dbachecks blog post – dbachecks – Setting temporary configuration values I have explained how this works so if you haven’t read it yet, take a look before continuing.
Ok, now that you know you can use the -Temporary to run the tests without interfering with the persisted, you may already notice what we will do next..
Disclaimer: First, let me say that this is just one option you can come up with a different one. Please drop a comment so I, and others, can become aware of different approaches.
Export-DbcConfig to save them.Set-DbcConfig) or import (using Import-DbcConfig) your configurations (set up on number 1) but don’t forget to use the -Temporary parameter.Invoke-DbcCheckAgain, take a look on your resources and then you can test with one more session. Do it until you find the sweet number of parallel sessions.
Here is the code you can use:
For 1st point:
#PROD environment Set-DbcConfig -Name policy.recoverymodel.type -Value Full -Temporary Export-DbcConfig -Path "D:\dbachecks\Prod_Configs.json"
#DEV environment Set-DbcConfig -Name policy.recoverymodel.type -Value Simple -Temporary Export-DbcConfig -Path "D:\dbachecks\Dev_Configs.json"
2nd, 3rd and 4th point together:
#PROD instances $sqlInstances = "prod1", "prod2", "prod3" #Import Prod_Configs.json with -Temporary Import-DbcConfig -Path "D:\dbachecks\Prod_Configs.json" -Temporary #Run the checks - Don't forget to add all the parameters you usually use Invoke-DbcCheck -SqlInstance $sqlInstances
#DEV instances $sqlInstances = "dev1", "dev2", "dev3" #Import Dev_Configs.json with -Temporary Import-DbcConfig -Path "D:\dbachecks\Dev_Configs.json" -Temporary #Run the checks - Don't forget to add all the parameters you usually use Invoke-DbcCheck -SqlInstance $sqlInstances
Save this scripts in two different ps1 files. Then, open two different PowerShell sessions and call each script on different session. Let it flow 🙂
On my case I was able to drop from 2 hours to about 1 hour with 3 parallel sessions. Adding a 4th session made the whole process slower.
We saw that we may have problems if we try to run more than one dbachecks session when using different configured values. Using -Temporary parameter when setting the values come in handy for this scenario.
This way we can run two or more sessions in parallel and each one on different environments without messing each other and hopefully, cut off our execution times.
Hope this helps! I would love to hear if you were able to drop down your execution times and what they are before and after.
Thanks for reading!
dbachecks has seen the light about two months ago. As I’m writing this blog post, the module counts with more than 2600 downloads just from the PowerShell gallery.
The module has about 110 configurable checks that make our live easier!
Today I will write about an option that I think users still do not realize that exists.
dbachecks works with the values previously saved (for that we use Set-DbcConfig). This means that when we start a new session and the last session has changed any configuration, that configuration is now, by default, the one that will be used in the new session.
Today I want to share a different option!
Let’s assume that you have your dbachecks configs set up for the Production environment. What do you need to do if you want to change just one check to test it in the Test environment?
One option is use the export/import method that Rob (b | t) wrote about on his dbachecks – Configuration Deep Dive blog post.
When we start a new session and we import dbachecks (in matter of fact when the PSFramework is imported – required module for dbachecks to work) we get the values from the registry. This means that we will read whatever is there at that moment.
-Temporary parameterThis parameter is available on Set-DbcConfig command. As said before, this command allows us to set a configuration which is, by default, persisted. But, if we use the -Temporary parameter we are saying that the configured value is only available for the current session the value will not be persisted for future executions, hence, will not mess with other new sessions.
You can run the following code to get the parameter description:
Get-Help Set-DbcConfig -Parameter temporary
Here is a demonstration:
This video shows that when we don’t use the -Temporary parameter and we start a new session we will read the last value set up. When we run the command with the -Temporary parameter (when setting the value to 5) after start a new session the value read will still be 3.
This way we don’t need to export/import the configurations. Perhaps this will save you time when doing some ad-hoc tests and not stay in doubt if you forgot to replace the older values after a different environment test with different configurations.
“But I already have and use the export/import method! Changing this can be more work…”.
We got that covered! 💪
If you run
Get-Help Import-DbcConfig -Detailed
you can see the -Temporary is also available in this command.
Hope this bring some new ideas like making your single, ad-hoc, one-time tests easier to configure!”
I have an idea that I will share on my next post about dbachecks!
-Temporary parameter exists on both Set-DbcConfig and Import-DbcConfig commands.
By using it, you are just changing the values on the current session and won’t overwrite the persisted values. This can become in handy in some cases.
Explore it!
Drop a message in the comments section either if you already use it and in which way or if you were not aware that it exists and will give it a spin!
Thanks for reading!
dbachecks is a new PowerShell module from the SQL Server Community! For more information, read introducing dbachecks.
If you don’t know dbachecks, we have released a good amount of blog posts that will help you:
Announcing dbachecks – Configurable PowerShell Validation For Your SQL Instances by Rob Sewell
introducing dbachecks – a new module from the dbatools team! by Chrissy LeMaire
install dbachecks by Chrissy LeMaire
dbachecks commands by Chrissy LeMaire
dbachecks – Using Power BI dashboards to analyse results by Cláudio Silva
My wrapper for dbachecks by Tony Wilhelm
Checking backups with dbachecks by Jess Promfret
dbachecks please! by Garry Bargsley
dbachecks – Configuration Deep Dive by Rob Sewell
Test Log Shipping with dbachecks
Checking your backup strategy with dbachecks by Joshua Corrick
Enterprise-level reporting with dbachecks by Jason Squires
Adding your own checks to dbachecks by Shane O’Neill
dbachecks – A different approach for an in-progress and incremental validation by Cláudio Silva
Other documentation:
dbachecks readme
dbachecks wiki (for developers)
I will share one of the ways I like to use dbachecks when I’m knocking down the problems in order to increase the green percentage and lower the red one!
How do you save the results?
Do you save one file per instance (all tests included)?
Using -Append?
Alternatively, one per check/environment?
There is not a single way of doing this. Neither a “correct way”.
Here you can find another different way grouping your results per application.
I will share the way I like to use it, when using the PowerBI dashboards to analyze the results, and explain the advantages I get from it.
My personal choice is to have one file per check and environment. This means that if I’m running a check for SuspectPage I run for all instances/databases belonging to the development environment, I will end with a file named dbachecks_1_SuspectPage_DEV.json.
Keeping the same line, I will get a filename dbachecks_1_SuspectPage_PRD.json if I run it for production.
$sqlInstances = "dev1", "dev2"
$checks = (Get-DbcCheck).UniqueTag
$checks.ForEach{
Invoke-DbcCheck -SqlInstance $sqlInstances -Checks $_ -PassThru -Show Fails | Update-DbcPowerBiDataSource -Environment "DEV" -Path "C:\windows\temp\dbachecks"
}
This will output:
“This will create a lot of files…”
Let’s imagine for a moment that we have to manage 3 different environments (DEV, QA, PRD):
Currently, we have 80 checks if your approach is 1 file per environment you will end up with 3 files. The way I like to do it, I will end up with 240 files.
WOW! Big difference right?
Yes, it is a big difference but that is no problem at all! The Power BI file will deal with this increase flawlessly as I have mentioned before on dbachecks – Using Power BI dashboards to analyse results blog post.
The biggest advantage, for me, is the possibility I have to re-run a single test for a single environment and with it, only touch just one of the files. It’s an update of that file.
By doing it, for the same destination folder, I will overwrite the existing file then I literally just need to go and hit “Refresh” button on PowerBI dashboards.
This way it took just the time of that test and not all of them. Quick and easily, I’m able to confirm that the fix I have run actually worked and my red values are lower! 😀
The point 6 is where you will save huge amounts of time because if you have just one file for all tests for one environment, you would need to rerun ALL the tests in order to refresh your environment.
Hope this helps!
Thanks for reading!
For the last couple of months, members of the dbatools team have been working on a new PowerShell module called dbachecks. This open source PowerShell module will enable you to validate your SQL Instances. Today it is released for you all to start to use 🙂
Here you can find a list of the blog posts made available today:
Announcing dbachecks – Configurable PowerShell Validation For Your SQL Instances by Rob Sewell
introducing dbachecks – a new module from the dbatools team! by Chrissy LeMaire
install dbachecks by Chrissy LeMaire
dbachecks commands by Chrissy LeMaire
dbachecks – Using Power BI dashboards to analyse results by Cláudio Silva
My wrapper for dbachecks by Tony Wilhelm
Checking backups with dbachecks by Jess Promfret
dbachecks please! by Garry Bargsley
dbachecks – Configuration Deep Dive by Rob Sewell
Test Log Shipping with dbachecks
Checking your backup strategy with dbachecks by Joshua Corrick
Enterprise-level reporting with dbachecks by Jason Squires
Adding your own checks to dbachecks by Shane O’Neill
dbachecks – A different approach for an in-progress and incremental validation by Cláudio Silva
Other documentation:
dbachecks readme
dbachecks wiki (for developers)
In this blog post, I will write about the Power BI dashboards that we have created to analyse the output of the tests.
Here, at dbachecks team, we don’t have BI gurus. This means, that what you are about to see come from our non-BI minds plus some research on the web to achieve the output we wanted.
The main objective is to have something functional, that helps to make decisions and help to know where we should look next.
That being said if you master Power BI, M query language, DAX or other stuff that we have used or can be used on this project and you found some archaic or a too much year 2000 way to do some stuff, let us know! We would love to follow best practices and improve the dashboards.
I started to work with Power BI after seeing a couple of sessions where it was and was not the main attraction. One of those sessions was from Rob Sewell (b | t) where he was showing Pester and, to show the output, he has used Power BI desktop. This means that he is the principal “culprit” for this choice 🙂. If, in addition to that, we join the ease of import the JSON (and other) files, the usefulness and the eye-candy dashboards all together made the decision easier.
With this in mind, I picked the initial Power BI from Rob and started digging on it and make some changes.
Note: If your tool of choice is not Power BI or you just feel more comfortable using another tool to work with JSON files, please share your dashboards with us. Write about it and share with the community.
dbachecks output consists of one or more JSON files. When you write your dbachecks scripts you can choose between just one or more files as result of your validation. For instance, you can generate one output file per context or just one that includes all contexts. You can even use the -Append to append results to an existing file as Rob described on dbachecks – Configuration Deep Dive blog post.
For a more pleasant and smooth experience, we recommend that you update your Power BI desktop to the latest version.
Nowadays, Power BI’s team releases a new version every month. Each new version brings new features and improvements. Because we may be using the most recent version of Power BI (desktop) to take advantage of some of those, we also encourage you to use the most recent version.
You can download it from the official Power BI website.
Note: If you try to open a Power BI file saved on a more recent version than the one you are using you may see an error message like:
This was the result of an attempt to open a file on the December 2017 version that was saved on a more recent (the February 2018) version.
dbachecks contains two different files:
By default, dbachecks will output the results to the “c:\windows\temp\dbachecks” folder. This folder is also the default one configured in the pbix file. This path is the only variable that we use on the pbit file, which means if you don’t need to change your path because you relied on the default one, you can just open the pbix file and click refresh.
On the other hand, the intent of the pbit file is to make it easy for people that are not so comfortable with Power BI desktop, this way they just need to type their output folder (where all the JSON files are) and they are ready to go.
Where this is different from the other file? On the other file (“pbix” extension), you need to go to edit queries menu in order to change your path. This is a more “advanced way” to do it, so if you just want to start to see your tests results in a beautiful way, you can open the template file (“pbit” extension), fulfil the requested path, hit Load and wait for it to load all of your results.
Note: When using pbit, if no objects appear, please make sure you have entered the correct path. Wrong paths will lead to empty dashboards.
The pbix file will keep the last data you saw on the dashboard. If you have rerun all or some tests, don’t forget to click the “Refresh” button!
Only after refreshing the data source you will get all the data (re) loaded and thus get the most recent version of it.
On this liftoff of dbachecks, our Power BI file includes two dashboards.
If you never used Power BI before you may not know that you can maximize just one visual (almost any type) and come back to the report whenever you want. This is cool when you have a lot of information and you want to take a closer look.
See this example (click to open on new window):
This dashboard gives to you a glance at how good/bad your environments are. This is done, at first, by the 2 circles with percentages and green/red waves.
On the right side you can find a matrix where you can drill-down from Environment -> Test -> Context. This drill-down can be hierarchical or absolute. To navigate on the levels, select the matrix and use the 3 buttons in the top-left corner of the visual (as shown on the last animation)
On the bottom, you can find a grid that shows all the failed (and only failed) tests with a descriptive message.
To help you to read this dashboard, on the left side you have the quick filters by Environment and Instance.
There are two other ways to filter the information – you can select a row on the matrix or on the grid to see information just related to that row.
Example:
The main objective of this dashboard is helping you to understand which tests take the most time to run.
Bear in mind that the times you will see are just the time that one test or the sum of tests took. The time switching between tests are not accounted.
I use this dashboard to understand which tests should/can be running in parallel and that way make the full execution faster.
Also, you can decide that some tests don’t need to run on the same frequency than others.
On the next example, I can see that my Production environment is the one taking the most time to complete, then I filter by it and I can see that the “Testing duplicate indexes” and “Testing Column Identity Usage” are the ones that take the most time. If I want, I can exclude them from the bar chart just to take a closer look at the other test times. Finally, I can go to the filters and remove them resetting the bar char right where we started.
Take a look:
Yes, there are some rules 🙂
To accomplish these dashboards, some rules must be followed. For example, the “Context” message needs to follow a specific nomenclature. You can read more about it on our wiki on Github.
If these rules are not followed when writing the tests, you can see some weird results on the dashboards.
On this example, you can see six instances, but the last two “Procedures” and “table” appear because the test wasn’t write in the right way.
Also, we have a unit test to help you check for these “Context” rules!
The data source on the Power BI was built to be dynamic. Because we can output just 1 test result per file (a record) or multiple results in the same file (a list), we built it so they can live together!
You may be thinking how fast it is to load the data and apply all the transformations we have to the files. We had that in mind and we tried to reach a good performance on that task.
I can load 270 files, totalling 397MB of data, in less than 30 seconds (the time will vary depending on the machine specifications).
If you are curious, each file contains a single test for every instance in a specific environment. At that time I managed 7 environments with more than 100 instances.
Now you can run your tests, analyse the output, make the changes needed, rerun the tests and start seeing your green percentage going up and the number of errors going down!
We will bring some new dashboards and improve the existing ones whenever possible.
If you have a case that you would like to see covered by a dashboard share with us. Do you already have it sort out? Share with us and we can replicate to our file.
dbachecks is open-source available on GitHub for anyone to contribute.
We would love you to contribute. Please open issues for new tests, enhancements, bugs. Please fork the repository and add code to improve the module. please give feedback to make this module even more useful.
You can also come in the SQL Server Community Slack and join the dbachecks channel and get advice, make comments or just join in the conversation.
I want to say thank you to all of the people who have enabled dbachecks to get this far. These wonderful people have used their own time to ensure that you have a useful tool available to you for free
Chrissy Lemaire @cl
Rob Sewell @sqldbawithbeard
Fred Weinmann @FredWeinmann
Stuart Moore @napalmgram
Shawn Melton @wsmelton
Garry Bargsley @gbargsley
Stephen Bennett @staggerlee011
Sander Stad @SQLStad
Jess Pomfret @jpomfret
Jason Squires @js0505
Shane O’Neill @SOZDBA
and all of the other people who have contributed in the dbachecks Slack channel
