Search This Blog

Friday, November 27, 2009

Sorts Spills in tempdb due to incorrect behavour by Query optimizer...

This one is really bothering me .I tried n tried n tried but failed to understand why is it happening ....

To give you a brief description , A query runs in 25-45 ms for x rows but the same query runs 15-20 times slower if i just add 1 row or a dozen more rows in some other caese (you will see it later)...If i see the execution plans , its totally identical .The only difference is that , the query which is slower does sorting in tempdb instead of in the user database where its been executing ...

Here we go :

create table SQLDBA (a1 int primary key clustered,a2 int,a3 char(2000))

begin tran
declare @i int
set @i=1
while @i <=200000
insert into SQLDBA values (@i,rand() * 2000000,replicate('a',2000))
set @i=@i+1
commit tran

update statistics SQLDBA with fullscan

So , we created a table with 3 columns and inserted 200000 rows to it .
Now I will run 2 queries :

set statistics time on

/* Good Query */
declare @a1 int,@a2 int , @a3 char(2000)
select @a1=a1 ,@a2=a2,@a3=a3 from SQLDBA
Where a1 <3000
order by a2

/* Bad Query */
declare @a1 int,@a2 int , @a3 char(2000)
select @a1=a1 ,@a2=a2,@a3=a3 from SQLDBA
Where a1 <3079
order by a2

Notice that there are only 79 more rows .But if you execute them on SQL Server 2005 SP3 as I did with 2 GB RAM , the 1st query runs in approax 15-22 secs .While the second query runs in 320-410 secs.Try it yourself .You will see some difference in duration as its hardware specific.Also , in my case it was running OK till 3078 rows . Yes, the execution time when a1 <3000 or a1 <3078 is almost similar .If I increase just 1 more row it degrades the performance .

NOTE : You might have to find out this threshold limit on your machine .In my case it was 3078 .The moment I use 3079 ..Booooommmmmmmmm

So why its happening :D ......

-Setting Statistics profile to ON showed that The Subtree cost is 2.5% more ..which is fine .

-Setting Statistics IO to ON showed :

Table 'SQLDBA'. Scan count 1, logical reads 1007, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

SQL Server Execution Times:
CPU time = 31 ms, elapsed time = 34 ms.
SQL Server parse and compile time:
CPU time = 0 ms, elapsed time = 1 ms.

Table 'SQLDBA'. Scan count 1, logical reads 1034, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

SQL Server Execution Times:
CPU time = 16 ms, elapsed time = 344 ms.

So , looks like this is Neither a CPU issue (query that runs faster is taking more CPU time) nor an IO issue as there are no physical reads.

One thing that I found when I took the XML output of the execution plan was that the good query was always taking compile time 2 and the bad query was taking 0 ....this was again strange to me.

Totaly stuck ..........

Then I searched many DMVs related to query execution and used sys.dm_exec_query_memory_grants DMV which show the row for each query that is waiting for the memory or those waited and acquired but will not show the querries that do not have to wait for the memory grant .

surprisingly , it was showing only one query in the output and that was the query taking more time .Strange but true .Why its waiting for memory when there is enough RAM on my server and max server memory is unlimited.

sys.dm_exec_query_stats also shows that bad query is doing around 100 physical reads while good one is doing none.

Then i queried sys.dm_exec_query_memory_grants:

select granted_memory_kb,used_memory_kb,max_used_memory_kb
from sys.dm_exec_query_memory_grants
where session_id in (51,52)

found that good query is using very less memory out of granted but bad query using full and more then that ........

Because whatever extra memory it needs for sorting is coming form tempdb :

select num_of_reads,num_of_bytes_read,num_of_writes,num_of_bytes_written
From sys.dm_io_virtual_file_stats(db_id('tempdb'),1)

So looks like Optimizer is not able to allocate the right amount of memory to the bad query such that it does not go to tempdb .......NOPE i am incorrect i think ..because if you see the number of rows in where condition the amount of memory given looks directly proportional .....S oallocating memory to the query does not seems to be a problem here ..the problem is why the query is using more memory that it needs to spill to tempdb .....

Let me give you one more example that will further open your eyes :D ...

set statistics time on
declare @a1 int,@a2 int , @a3 char(2000)
select top 100 @a1=a1 ,@a2=a2,@a3=a3 from SQLDBA
Where a1 <3000
order by a2


declare @a1 int,@a2 int , @a3 char(2000)
select top 101 @a1=a1 ,@a2=a2,@a3=a3 from SQLDBA
Where a1 <3000
order by a2

Now , these queries are identical in any respect except that in the second query I am doing select top 101 ...just 1 row extra ....and see the difference ...HELL ....
And the status is same here if you go through the DMVs etc ..Even if you wrap up in a SP or use different hints wont help ...

However you will see that In this case i.e. the second one 1 MB of memory is given to both the queries and nonoe of them is consuming the complete memory given ...but still the bad query is doing Sort spills to tempDB .....So its proved that granting of memory to the queries by the optimizer is Correct ...but its the way query sorts in tempdb and also the usage of memory by the query that looks incorrect here ...

So whats happening ........ :D ...Why just one extra row makes the query to spill to tempdb for sorting ....(use profiler to see if sorting in happening)..


On my machine i have 2GB RAM .....i capped max server memory to 100MB...
And then the query was running fine top 100 and top 101 and the previous query all are running well except that the bad query takes more time once in 7-8 attampts (sometimes less or sometimes more)....

Looks like something is related to memory regions or chunks ...not sure , I may be incorrect ....Something is really wrong .....

Filing a bug ..

Happy learning ...

Friday, November 20, 2009

Creating SQL Server Alerts using WMI classes- Part 1

When we create alerts or when we try to create script we search net a lot .We get some thing but we do not get all what we look for .And then we stop pursuing it ...

I was given a task to create SQl Server alerts (as many as I can).I could find a couple of scripts ready but that did not satisfy my requirement as everyone creats the scripts as per their needs .

So I decided to do everything on my own and hence sharing my experiance with you .
To know what kind of alerts you can create go through this file

You will be amazed to see that almost everything is covered here :-) ..

Let us start now by creating an alert that will notify us whenever a new database is created .

Pre-requsites :

-> Need to be SQL Server 2005 SP2 or above or SQL Server 2008 as there is a bug in RTM or SP1 .

-> We need to execute it in master (do not change the script to be run in your user database).This is because Master captures the metadata of all the databases including itself .If you create it in anyother database then it will not work .Other scripts will also work only for that user database .

-> Make sure the SQL Server Agent is started and make sure that "REPLACE TOKENS FOR ALL JOB RESPONSES TO ALERTS" option is checked under Agent properties -> Alert system .

-> Read the code a few times and understand it .You might have to change it as per your need .

Part 1:

* This script will create an Alert to Monitor Create DATABASE event
* The alert will run a job and the job will enter data in a table.

/* Step 1: creating the table to capture the Event information */

USE Master

IF EXISTS (SELECT * FROM dbo.sysobjects WHERE id = OBJECT_ID(N'[dbo].[Create_database]') AND OBJECTPROPERTY(id, N'IsUserTable') = 1)
DROP TABLE [dbo].[Create_database]

CREATE TABLE [dbo].[Create_database] (
[PostTime] [datetime] NOT NULL ,
[Instance] varchar(20),
[DatabaseName] varchar(20),
[computerName] Varchar(20),
[Loginname] Varchar(20),
[TSQLCommand] varchar (1000),
[Flag] [int] NOT NULL CONSTRAINT [DF_Create_database_Flag] DEFAULT ((0))

CREATE INDEX [Create_database_IDX01] ON [dbo].[Create_database]([Posttime]) WITH FILLFACTOR = 100 ON [PRIMARY]

/*Step 2 : Creating the Job that will enter values into the table created above*/
/*Service account and sql operator option are optional and hence commented*/

USE [msdb]

IF EXISTS (SELECT job_id FROM msdb.dbo.sysjobs_view WHERE name = N'Capture Create_Database Event')
EXEC msdb.dbo.sp_delete_job @job_name = N'Capture Create_Database Event', @delete_unused_schedule=1

--DECLARE @ServiceAccount varchar(128)
--SET @ServiceAccount = N''
--DECLARE @SQLOperator varchar(128)
--SET @SQLOperator = N''

SELECT @ReturnCode = 0

IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N'[Uncategorized (Local)]' AND category_class=1)
EXEC @ReturnCode = msdb.dbo.sp_add_category @class=N'JOB', @type=N'LOCAL', @name=N'[Uncategorized (Local)]'
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback

EXEC @ReturnCode = msdb.dbo.sp_add_job @job_name=N'Capture Create_Database Event',
@description=N'Job for responding to Create_database events',
@category_name=N'[Uncategorized (Local)]',
@job_id = @jobId OUTPUT

IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback

/*Step 3: Insert graph into LogEvents*/

EXEC @ReturnCode = msdb.dbo.sp_add_jobstep @job_id=@jobId, @step_name=N'Insert data into LogEvents',
@os_run_priority=0, @subsystem=N'TSQL',

INSERT INTO Create_database (


IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_update_job @job_id = @jobId, @start_step_id = 1
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback
EXEC @ReturnCode = msdb.dbo.sp_add_jobserver @job_id = @jobId, @server_name = N'(local)'
IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback

GOTO EndSave


/*Creating the alert and associating it with the Job to be fired */

USE [msdb]

IF EXISTS (SELECT name FROM msdb.dbo.sysalerts WHERE name = N'Respond to Create_database')
EXEC msdb.dbo.sp_delete_alert @name=N'Respond to Create_database'


DECLARE @server_namespace varchar(255)
SET @server_namespace = N'\\.\root\Microsoft\SqlServer\ServerEvents\' + SUBSTRING(@@SERVERNAME, ISNULL(CHARINDEX('\', @@SERVERNAME), 0) + 1, LEN(@@SERVERNAME) - ISNULL(CHARINDEX('/', @@SERVERNAME), 0))
SET @server_namespace = N'\\.\root\Microsoft\SqlServer\ServerEvents\MSSQLSERVER'

EXEC msdb.dbo.sp_add_alert @name=N'Respond to Create_database',
@notification_message=N'Your Message',
@wmi_query=N'SELECT * FROM Create_database',
@job_name='Capture Create_Database Event' ;

EXEC msdb.dbo.sp_add_notification @alert_name=N'Respond to Create_database', @operator_name=N'Test', @notification_method = 1

--select * from master..Create_database

How it works :

As soon as the database is created WMI class create_database will generate an event and trigger the alert through SQL Server agent .The alert will in turn execute the job that will insert the event information to the table we have created .

Part 2:
This part will send a mail to the desired user that an alert has been fired .
This can be achieved by :

1) Using Database mail (including creating an operator)and choosing the option in the job itself for notifying the operator .

2) Using custom based CDO message via VBscript and execute it using XP_cmdshell as a job step.

Mail.vbs :
Set objEmail = CreateObject("CDO.Message")
objEmail.From = ""
objEmail.To = ""
objEmail.Subject = "Mail for you using CDO :-) "
objEmail.Textbody = "Hi sir this mail has been sent without using SQL Server and from WMI script."
objEmail.AddAttachment "C:\Script repository\Alerts\final Alerts\readme.txt"

3) Using the Stored procedure below :
/* Step 5: Create a stored proc for sending the [Create_database] information as .CSV file */

Create proc [dbo].[Create_database_rpt]
DECLARE @SQL varchar(2000)
DECLARE @date varchar (2000)
DECLARE @File varchar(1000)
select @date= convert(date,GETDATE())
SET @SQL = 'select * from [Create_database] where flag = 0'
SET @File = 'Deadlock report'+@date+'.csv'

EXECUTE msdb.dbo.sp_send_dbmail
@profile_name = 'test',
@recipients = 'your',
@subject = '[Create_database] report',
@body = '***URGENT***Attached please find the Create Database report',
@query =@SQL ,
@attach_query_result_as_file = 1,
@query_attachment_filename = @file,
@query_result_header = 1,
@query_result_separator = ' ',
@query_result_no_padding = 1,
@query_result_width = 32767

/* Step 6: Changing the flag to 1 so that next time this information is not sent*/
update dbo.[Create_database] set flag = 1 where flag = 0

Like this Try your hands on different alerts by going through the classes copied above .You can also go through the classes at MSDN :

Hope this helps .I am also learning ropes on WMI and will post some good stuff as soon as I am ready .Keep reading as there is much more to come .

Happy learning .

Msg 3132, Level 16, State 1, Line 1

Today as soon as i reached office in the morning , an issue came up which was related to restoring of SQL Server database .The issue was that we were not able to restore one of the user database backups taken on some remote server .

The error we were getting was :

Msg 3132, Level 16, State 1, Line 1
The media set has 2 media families but only 1 are provided. All members must be provided.
Msg 3013, Level 16, State 1, Line 1
RESTORE DATABASE is terminating abnormally.

There was some .bak say test.bak file, which we were trying to restore and getting the error above.

This error says that there are 2 media families and only 1 has been provided .
Media families here means a Disk or a tape .If we take it as disk , then it means that "The backup was taken on two backup devices or say backup files (in case you have not created disk devices)" . This kind of backup is called as a srtipped backup where both the backup devices have some part of backup but not complete.

But the guy who took the backp says that he did nothing special that made the stripped backup i.e. he did not take the stripped backup at all .

Then how come the stripped backup took place on it own ....this was pretty strange.

I checked this issue through Bing and google and most of the people were saying the same that they never did intend to take the stripped backup .

Now, i had to prove the client that we are missing one backup device here or we are missing one .bak file , I did a small repro :

USE master
EXEC master.dbo.sp_addumpdevice
@devtype = N'disk',
@logicalname = N'rest_issue',
@physicalname = N'C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Backup\rest_issue.bak'

EXEC master.dbo.sp_addumpdevice
@devtype = N'disk',
@logicalname = N'rest_issue_1',
@physicalname = N'C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Backup\rest_issue_1.bak'

backup database testdba to rest_issue,rest_issue1
restore database testdba from rest_issue <-- used only one mediaset

Msg 3132, Level 16, State 1, Line 1
The media set has 2 media families but only 1 are provided. All members must be provided.
Msg 3013, Level 16, State 1, Line 1
RESTORE DATABASE is terminating abnormally.

restore database testdba from rest_issue,rest_issue1 <-- used both mediaset

Restore was successful..
So the error was genuine and working fine .

Further ,If you think that you will restore only 1 file i.e. data file and leave the log file using option partial , even that will not work as the backup for data file is also stripped.

Restore filelistonly will tell you the files in the backup set

Restore headeronly will give us the MediasetID number and other information

Restore labelonly will give us the BindingID number and other information

Restore verifyonly will fail with the same error if you use it on one file as it needs both the files

Solution :There is no solution to this issue you need to find out another device else it will not work (use restore headeronly and labelonly ,backupmediafamily, backupset for assistance) and use both the files to restore .Once you have it , you can issue this

command :
Restore database ibmdba from anoop_issue,anoop_issue1 with replace
OR Restore database ibmdba from disk = , disk = with replace

Why Hhis happens :)

This will happen if :

1) You have created two(or more) backup devices and taken backup on 2
(or more) devices or you can say stripped backup and while restoring yo uare using only one .

2) You are using GUI to take the backup without seeing that the backup is going to more than 1 device or files .If you select both the devices/files it will take stripped backups .If you insert 2 devices / files but select NONE of them then also it will happen (That should not happen and should give us the error "Please select a device") .And thats where the bug is .

So the BUG is :

If a user do not select a device in the backup set when more than one devices are existing , by default SQL Server selects all the devices and not giving the error "please select the device"....

If this default behaviour is by design then its not the correct design and should be changed .

Repro of this issue through GUI :
-> right click adventure works database > Tasks > backup
-> Under Destination remove anything if its mentioned by default.
-> Click on Add > a new box will pop up > select backup device you have created
-> you will get the device name > click on OK.
-> Again click on Add >a new box will pop up > select another backup device .
-> You will see that both the backup devices are unselected now (",).
-> If now you click ok it will take the backup .Ideally it should give the error that "please select the device".

I have filed a bug with MS through connect...

Happy learning