| JOURNAL HOME | CME HOME | THIS MONTH | PAST ISSUES | ETOC | COLLECTIONS |
| AUTHORS | REVIEWERS | EDITORIAL BOARD | FEEDBACK | RSS | HELP |
 
|
|
|
||
|
|
Anesth Analg 2009; 108:929-940 © 2009 International Anesthesia Research Society doi: 10.1213/ane.0b013e3181921c37
Section Editor: |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
METHODS
Appendix 1 describes the mathematics programmed using structured query language (SQL Server, Microsoft, Redmond, WA). That Appendix is the part of the article that describes the underlying mathematical basis of the intervention. The Appendix is presented in sufficient detail that other facilities can implement our methods.
Every 5 min, a stored procedure on the AIMS database server calculates the elapsed time in each case from patient entry into the OR, for every case currently running in all 29 ORs of four physically distributed suites. The OR director's status display (whiteboard) and the managerial decision support system are updated automatically with these times. The time of patient entry into each OR is obtained from the AIMS (see Discussion).9 The surgeon and scheduled procedure(s) are obtained from the hospitals' operating room information management system (ORIMS). Those data are used to choose parameters from lookup tables that are regenerated using a SQL procedure that is run quarterly using all data from January 1, 2001 to the current date. The Results presented here are based on cases from 2001 to 2007 applied to forecasts for February and March 2008.
The Bayesian method to estimate OR times is applied regardless of whether the case is of a combination of surgeon and procedure(s) with no, a few, or many historical cases of the same type.8 When a case is of a rare combination with little or no historical data,10,11 the scheduled OR time is the principal or sole basis for future predictions of OR time. When a case has substantial historical data, the scheduled OR time has a negligible effect on OR time as compared to the information from the historical OR times. At the studied hospital, the schedulers add an estimate of the turnover time to the estimate of the OR time provided by the surgeon. Therefore, the scheduled OR time used in the Bayesian method is estimated from the scheduled case duration (see Appendix 2).
Equation 11 in Appendix 1 gives the automatically updated estimate of the time remaining in each case. However, the Bayesian method only uses data available before the case started and the automatically determined7,9,12 time from case start. The longer that each case proceeds, the greater is the expected incremental value of the knowledge of the providers in the OR. Therefore, occasionally an instant message is sent to the AIMS workstation to elicit an update from the anesthesia provider of the remaining OR time (Fig. 1) (Appendix 3). The elicited time replaces the scheduled OR time for status displays, the decision support system, and future elicited times. Appendix 4 describes how we used the data from 560 such cases to assess the "up-time" of the system and how long anesthesia providers took to respond to the dialogs.
RESULTS
Figure 2 shows an example of how the predicted OR time remaining in cases differs among cases depending on the parameter values of the combinations of surgeon and procedure(s). Both lines are for cases with a scheduled OR time of 1 h and a median of historical OR time of 1 h. After 1 h in the OR, the updated median time remaining for the two cases are 9 min and 27 min, respectively. Later (e.g., at 1.5 h in the OR), the median times remaining are persistently close to a constant of 7 min (12% of 60 min) (– – –) or 28 min (47% of 60 min) (—), respectively. For the latter case (—), if the surgeon was asked at 1.5 h "how much longer," a likely answer would be "a half hour." At 2.0 h, if the case were still ongoing, a likely answer would still be "a half hour." Figure 2 shows that this commonplace opinion about surgeons' updated estimates is reasonable, but that the additional OR time that is needed differs among cases. See the last three paragraphs of Appendix 2 for statistical details.
|
Figure 3 expands upon the result of Figure 2 by showing results for the estimated parameters of 560 cases. The cumulative frequency distribution on the left shows that, among cases that exceed the 90th percentile of their predicted OR time distribution, the median time remaining varies considerably among combinations. The 10th, 50th, and 90th percentiles are 16 min, 35 min, and 86 min, respectively. The cumulative frequency distribution of the times remaining each expressed as a percentage of the median of the predictive distribution is shown in the graph on the right. The 10th, 50th, and 90th percentiles are 12%, 21%, and 38%, respectively. The importance of the heterogeneity among surgeons and procedure(s) shown by Figure 3 is that automatic estimation of remaining OR time needs to rely on historical case duration data for each combination (i.e., Eqs. 1–11). It is inaccurate to consider all cases that have exceeded their scheduled durations as having either 30 min or 20% of their scheduled durations as their times remaining, or any other constant for all cases at a facility.
|
Figure 4 shows median times remaining as a function of the elapsed time for all cases of the two most commonly scheduled durations (see caption). Analysis is nonparametric (i.e., does not rely on the Bayes method and its statistical assumptions). As each case progresses, the average time remaining declines, remains moderately constant, and then may increase. However, increases occur once cases have been on-going so much longer than scheduled that there are few such cases. Thus, the 95% confidence intervals (CI) for the median are wide.
|
Every 5 min, the SQL implementation of the Bayes approach updates the estimated times of every case at the 29 OR surgical suite that is ongoing or not yet started. The under-estimate for the up-time of this process of automatic calculation of estimated times was 99.3% (95% CI: 98.1%–99.9%) (n = 33 days). Equivalently, for >119 of each day's 120 5-min intervals, the display was up to date for every case that was on-going or had not yet started. Appendix 4 has more details.
Appendix 5 describes some observations and resulting design decisions. Although anesthesia providers are physically present in ORs, can observe the actual progress of cases, and ask surgeons for feedback on times remaining, elicited estimates were only 4.7 min more accurate than the Bayesian method. In addition, the "instant message" dialogs are negotiated interruptions (see Appendix 4). Thus, times remaining are elicited only for cases that have exceeded their scheduled durations by more than 13 min. Following Sandberg et al.13 in their use of AIMS prompts, we made no announcements, sought no buy-in, and performed no education about the new system. The percentiles of latencies14 to acknowledgment were 1.2 min (50th), 2.1 min (75th), and 6.5 min (90th). The corresponding 95% CI were 1.1–1.3 min, 1.9–2.2 min, and 4.4–7.0 min, respectively (n = 560 cases).
DISCUSSION
Currently, at the studied hospital, the OR directors' status displays (whiteboard) and the managerial decision support system are updated automatically, without relying on clinicians for data entry. Every 5 min, Bayesian estimates are recalculated for every case that has either not yet started or is on-going. When instant message dialogs (Fig. 1) are sent (e.g., if resulting decisions are ambiguous), those elicited responses replace the scheduled OR time in the Bayesian method for future whiteboard updates (i.e., if a person estimates 45 min left at 4:30 pm, the boards do not incorrectly display 15 min left if the case is still on-going at 5:00 pm). Figures 2–4 capture the principal results of our article, which show the need for those updates.
Since status displays communicate15 publicly the valued16 work of individuals and the information affects decisions (e.g., assignment of add-on cases),2 prior ethnographic results apply showing that some people manipulate15 the data displayed when given the opportunity. For example, they may indicate that the case will last longer than expected to reduce the chance of the late afternoon add-on case from the intensive care unit being assigned to their OR. Automatic updating is unbiased.
Implementation of the automatic updates of displays (whiteboards) and the decision support system relies on knowing what case are underway in each OR at all times. We automatically infer the actual location of cases based on the identifier of the AIMS workstation transmitting pulse oximetry, electrocardiogram heart rate, and end tidal CO2 partial pressures to infer the location of each case, as previously described.9 That information is used to identify the historical data for each case to apply the Bayesian method (Figs. 2 and 3).
The statistical method is applicable for every surgical case, whether the case has 0 historical data, 99 prior data, or a value in between. That matters, because many cases at both tertiary and outpatient facilities have few or no historical data (e.g., due to rare procedure(s) and new surgeons).10,11,17,18 Furthermore, increasing the precision of case duration estimates depends on using all of the relevant data in OR information systems (e.g., not just surgeon and scheduled procedure(s), but also type of anesthetic and surgical team).19 The result of the effort to increase precision is fewer historical data per case.20
There are three principal limitations to our work.
First, recommendations from decision support systems improve decision-making on the day of surgery.21 In contrast, decisions made using status displays of ongoing OR cases, without recommendations, are correct at a rate no better than by chance alone.21 The accuracy of estimates of times remaining in cases has no substantive effect on over-utilized OR time or on overtime resulting from the decision support system's recommendations, because rarely is the inaccuracy of a sufficiently large magnitude to affect recommendations.1,2,22–24 Thus, how the estimates of times remaining are calculated is likely unimportant compared to the calculations being done systematically and automatically for every case. We could not, however, examine differences in resulting decisions, because anesthesia providers' estimates for time remaining are for single instants in time, whereas Bayes estimates are updated continually.
Second, the "instant message" negotiated interruptions from the dialogs and the automatic (Bayesian) methods are alternatives to using the telephone. Surgical cases studied at a facility averaged 14 and 20 interruptions per case, 1.5 by telephone.5,6 The telephone results in an immediate interruption (see Appendix 4) and its elicited time then needs to be typed into the computer for display or use in decision support. However, we do not know whether the negotiated interruptions, or the automatic system's elimination of interruptions, benefited OR team performance.
Third, status displays are useful not only because they contribute to decision support, but also because they facilitate asynchronous communication.3,7,15,25 OR coordinators use them to show the fairness of their decisions.25 Information is available so that others can provide suggestions.3,15 Staff know that supervisors and colleagues are aware of their activities.15 Anesthesiologists know when cases that they are medically directing will soon end.7 Surgeons can obtain information about when to-follow cases may start.7 However, we studied the automatic updating of displays, not how they are used. We do not know how automatic updating may be influence this use of the displays.
In summary, we successfully implemented automatic electronic whiteboard updates at a large surgical suite and have been using it to date (January 2009). We did this by deriving the Bayesian lower prediction bounds of case durations conditional on the minutes that the cases have been on-going. Every 5 min, estimates of OR times for cases that have not yet started or are underway are updated automatically for their status displays (whiteboards) and decision support systems. The system is useful, because after a case has been in the OR substantially longer than scheduled, the median expected remaining time is relatively constant, but with the remaining time differing substantially among surgeons and scheduled procedure(s).
Our notation matches that which we used previously to describe,8 validate,8 and apply8,26 the Bayesian method. Let the random variable Xk, refer to the natural logarithm of the OR time of a single case that is classified19 by its being of the kth combination of surgeon and scheduled procedure(s), k = 1, 2, ..., p. For example, k = 1 might be Dr. Smith scheduled to perform bilateral myringotomy tube placement and adenoidectomy in a child (i.e., Current Procedural Terminology-4 codes 69421 for one ear, 69421 for the other ear, and 42830 for adenoidectomy). We henceforth refer to each of the p combinations as "surgeon and procedure(s)". The nk previously observed (historical) OR times for the kth combination of surgeon and procedure(s) are exp(xk1),exp(xk2), ..., exp(xknk). For example, the n1 = 87 historical OR times might be 50 min, 30 min, ..., 40 min, resulting in the log transformed values of x11 = 3.9, x12 = 3.4, ..., x1n1 = 3.7. The sample mean of the nk historical data xk1, xk2, ..., xknk equals 
k, and the sample variance equals 
k2. The corresponding scheduled OR times in hours are exp(xsk1), exp(xsk2), ..., exp(xsknk). We assume that Xk follows a normal distribution with an unknown mean and variance. Our objective is to predict the OR time of the next case, exp(Xk*) based in part on its log scheduled OR time, xsk*, a constant known before the case begins. The asterisk symbol (*) is used to represent the next case.
The probability distribution for the next occurrence of an event after having observed historical data is called the posterior predictive distribution. We derived in Ref. 8 that the posterior predictive distribution of Xk* before the case begins follows (
) a scaled Student t-distribution with 2
k* degrees of freedom:
|
|
where
|
|
|
|
|
|
The constants and β are the parameters of an inverse 
distribution that represents the prior distribution of the population variance of Xk among the p combinations of surgeon and procedure(s),8 and has previously been shown to provide a good fit (e.g., see Fig. 1 of Ref. 8). The constant 
represents the ratio of the population variance of Xk to the variance of the prior distribution of the population mean of Xk.
Equation 2 gives the Bayesian weighted median predicted OR time of the next case. When a case is of a rare combination with little or no historical data,10,11 the scheduled OR time is the principal or sole basis for future predictions of OR time, since
|
|
and
|
|
When a case has substantial historical data, the scheduled OR time has a negligible effect on OR time as compared to the information from the historical OR times, since
|
|
and
|
|
.Equation 2 shows that represents the number of historical cases, of a given combination of surgeon and procedure(s), for which the median predicted OR time of the next case is based equally on historical case duration data and the scheduled OR time.
The 100 x gth percent lower prediction bound for the duration of each case, dg, is given by:
|
|
for 0 < g < 1. For example, when g = 0.05, there is a 5% chance that the OR time of the next case will be briefer than the value of dg. This is known as the 5% lower prediction bound for the duration of the next case.2,8,27 From Eq. 1, this bound satisfies the expression8:
|
|
where T[.,2k*] is the cumulative distribution function of the Student t-distribution with 2k* degrees of freedom and T–1[g, 2k*] is its 100 x gth percentile. For example, for g = 0.05, T–1[0.05, 2k*] = –1.65 for large k*. When g = 0.05, there is a 95% chance that the OR time will be at least as long as the value of Eq. 6. This lower 5% prediction bound is almost always less than the scheduled OR time and is useful for deciding when the next patient in the same OR on the same day needs to be ready.2,8,28 A low percentage is used to mitigate delays due to patient unavailability.
We previously published an empirical study of the accuracy of Eq. 6.8 For g = 0.05, the result d0.05 exceeded the actual duration of 4.9% of the studied cases.8 For g = 0.90, d0.90 was exceeded by the actual duration of 9.7% of cases.8 The absolute errors of the expected values averaged 3.0 min less than the absolute errors of the scheduled OR times (P < 0.0001). The term under the square root of Eq. 6 expresses the uncertainty in the prediction. As we address at the start of the Results and at the end of Appendix 2, the term varies substantively among combinations of surgeon and procedure.8
We next expand upon our prior work.8 Consider a case that has been on-going for time dmin > 0. The 100 x gth percent conditional lower prediction bound dg|dmin satisfies
|
|
From the definition of conditional probability and the fact that dg|dmin 
dmin, we find
|
|
Combining Eqs. 7 and 8,
|
|
Substituting Eq. 1 into the right-hand side,
|
|
where,2,28
|
|
|
|
and 100 x pk|dmin is the expected percentage of cases with the same parameters (Eqs. 2–4) that would have ended by time dmin. Applying Eqs. 5 and 6, the 100g percent conditional lower prediction bound is given by the 100
percent lower prediction bound for the duration of the case before it started:
|
|
For example, there is a 50% chance that the OR time of a case that has been on-going for dmin units of time will be smaller than the value d0.50|dmin in Eq. 11.
Figure 2 shows two examples of d0.50|dmin as a function of dmin. There are four reasons why we use g = 0.50 (i.e., the median time remaining). First, using the median minimizes the mean absolute error.17 Second, when provided anchored values, participants in experimental studies provide unbiased estimates for the median.29 Unbiased estimators have the property that the sum among cases of the differences between the actual and estimated values equals zero (i.e., there is no bias). Third, over 3 yr at the hospital with previously published results, schedulers who were supposed to estimate the OR times of cases did so by providing unbiased estimators for the median OR times.8 Fourth, the method is simpler to explain because the initial estimate is simply exp(µk*) from Eq. 2.
APPENDIX 2: IMPLEMENTATION OF BAYESIAN METHOD
OR information system data at the studied hospital were available from January 1, 2001 onwards. We created a lookup table of nk, k, and the sum of squared terms in βk*, while limiting consideration to the most recent 99 cases27 of each combination of surgeon and procedure(s). However, the historical (2001–2007) raw data contained obvious errors (e.g., negatively valued or missing OR times). Therefore, when setting up the table, we excluded cases for which the actual OR time divided by the scheduled case duration was <0.25 or greater than 4.0. Using the 129,380 historical cases, the criteria corresponded to trimming data 3.1 and 3.3 standard deviations below and above the mean difference, respectively. The numbers of cases excluded were 416 cases (0.3%) for missing (zero valued) actual or scheduled times, 546 cases (0.4%) with ratios <0.25, and 362 cases (0.3%) with ratios greater than 4.0. The histogram of the difference of the logarithms of actual and scheduled OR times in Figure 5 uses 99.0% of all cases. The information from these 128,056 cases from 2001 to 2007 was used to obtain the forecasts for the Results (from February and March 2008). However, in practice, the lookup table is recalculated using a SQL server procedure that is run quarterly using all data from January 1, 2001 to the current date.
|
The nurse manager of the OR information management system reported that the schedulers add an estimate of the turnover time to the estimate of the OR time provided by the surgeon. If an estimated case duration is not provided by the surgeon, the scheduled duration that they use is the trimmed mean of the most recent 15 historical OR times for the surgeon performing the procedure, plus an estimated turnover. We therefore needed to analyze the resulting scheduled case durations and actual OR times to determine how to estimate scheduled OR times (i.e., time in the OR for Eq. 2) from scheduled case durations (i.e., that from the preceding paragraph). From among the 128,056 cases, there were 8045 cases (6.3%) that were combinations of surgeon and procedure(s) observed only once (i.e., singletons) and 3370 (2.6%) only observed twice (i.e., doubletons). The scheduled OR time was set at 96.3% of the scheduled case duration, because doing so resulted in an unbiased estimator for the 50th percentile of OR time for these 11,415 cases (for details see Limitations section of Ref. 8). We limited focus on these cases because the impact of the scheduled OR time is progressively less in Eqs. 2 and 6 for each increase in nk. To check our use of the scheduled OR times, we then limited consideration to the 560 singletons and 470 doubletons for which other surgeons had scheduled the procedure(s) at least 30 times. The mean OR times from the other surgeons were used as the estimate of the OR time for the singletons and doubletons.2,30 The mean ± se of the absolute error was just 2.9 ± 1.4 min less than for use of the resulting scheduled OR times. As for the previously studied hospital, use of scheduled OR times was essentially no worse than use of other surgeons' times, but offered the advantage of there being a value for every case.8
To estimate the and β in Eqs. 1–11, we followed the approach described in the Appendix of Ref. 8 The actual OR times were obtained and their logarithms were calculated for the 50,559 cases of 773 combinations of surgeon and procedure(s) with at least 30 cases. The sample variance was determined for each combination. The inverses of the 773 sample variances were fit to a distribution using Systat 12 (SYSTAT Software, San Jose, CA). From the method of moments, the shape parameter was 2.33 and the scale parameter was 5.68, giving estimates = 2.33 and β = 0.176. Those values are close to the = 2.32 and β = 0.142 published from another hospital.8
Application of the estimated was an implementation challenge. The decision support system repeatedly calculates percentiles from t-distributions with noninteger degrees of freedom (i.e., 2k* in Eq. 6).2,8–11 We wrote the SQL code to calculate the cumulative distribution function of the t-distribution using its relationship with the Incomplete Beta Function.31 The inverse is obtained iteratively using bisection over a bracketed interval from values stored in a lookup table. For example, for g =0.26 and 2k* = 5.72, we aim to estimate T–1[0.26, 5.72]. We know a priori that T–1[0.25, 5] 
T–1 [0.26, 5.72] T–1[0.30, 6]. The lower and upper bounds both are stored in the lookup table and are used as the limits of the bracketed interval.
To estimate the , we again used the actual and scheduled OR times for the 50,559 cases of combinations of surgeon and procedure(s) with at least 30 cases. The sample variance of the differences of the logarithms of the actual and scheduled OR times equaled 0.158. The variance of a t distribution with 2 degrees of freedom equals32
|
|
. Setting nk = 0 in Eq. 2, resulting in µk* = xsk* in that equation, and substituting into Eq. 1, Var(Xk* – xsk*) =
|
|
=
|
|
. Solving the equation Var(Xk* – xsk*) = 0.158 for leads to = 5.14 cases. In comparison, we found = 8.68 cases at the previous hospital studied.8 The accuracy of the Bayesian estimates was insensitive to these relatively small differences in the value of (see last section of Appendix of Ref. 8). Estimation of was unaffected by the proportional correction for bias, because the correction shifted only the mean of (Xk* – xsk*), not the Var(Xk* – xsk*).
Figure 2 shows an example of the impact of differences in parameter values among surgeons and procedure(s) on the relationship between d0.50|dmin and dmin. For simplicity, we set µk* = ln(60 min). We used nk = , and consequently k* = + nk/2 = 2.33 + 5.14/2 = 4.90. For βk*, we used the two values 0.18 and 1.27, which corresponded to the 10th and 90th percentiles of the distribution of βk* for cases with 0 nk 9.
Figure 2 can be interpreted as a survival or reliability analysis. The bottom pane shows that after the patient has been in the OR for exp(µk*) time, the median residual lifetime is nearly constant. Let
|
|
. For βk* = 0.18, 
k* = 0.20. For βk* = 1.27, k* = 0.53. Generally, the hazard rate of the lognormal distribution increases to a peak and then declines.33,34 However, for our range of k*,18 the time of the peak in the hazard rate substantially exceeds the median OR time.33,34 Consequently, for times from exp(µk*) to exp(µk*), the hazard rate usually increases to a peak,34 and if it declines that is for a vanishingly small percentages of cases. The hazard rate can be examined using a statistics packages by generating log normally distributed random numbers with parameters µk* and k*, and then applying the package's parametric (lognormal) or nonparametric survival analysis.
There were n = 560 cases studied as part of Appendices 3 and 4. We used the estimated posterior parameters (k*, βk*, nk, and µk*) from these cases to explore simpler (non-Bayesian) estimation of the time remaining in surgical cases (Fig. 4). In addition, we used the estimated posterior parameters to generalize Figure 2 to all cases (Fig. 3). For each case, the 90% lower prediction bound (d0.90) was calculated using Eq. 6 and then substituted into Eq. 11 to calculate the median time remaining conditional on the case already having taken as long as its 90% lower prediction bound (d0.50|d0.90). Figure 3 shows cumulative histograms reporting the difference between d0.50|d0.90 and d0.90 and the difference reported as a percentage of d0.50.
APPENDIX 3: "INSTANT MESSAGES" AND THEIR IMPLEMENTATION
The mean time from the end of surgery to exit from the OR was determined from an analysis of the 6 yr of historical data in the hospital's ORIMS. Conveniently, the mean was 15 min. Every 5 min, a SQL stored procedure calculates the elapsed time in each case in progress from patient entry into the OR. If the elapsed time exceeds the scheduled case duration plus 10 min, and the "end of surgery" event has not been documented by the anesthesia provider, the stored procedure writes a message, addressed to the recipient workstation, to a database table. Each workstation queries the database at 1 min intervals and displays new messages directed to its attention using the dialog shown in Figure 1. This process resulted in a mean time from the end of surgery to the message of 12.5 min, since the query interval was 5 min. Because workstations queried the database for messages every 1 min, there was an additional mean latency of 0.5 min. Together, the earliest that the message could be sent was 10 min after the scheduled case duration, the latest was 16 min, and the mean was 13 min.14
When asked, respondents use a digital clock to respond with an estimate of the time when the patient will exit from the OR (Fig. 1). The clock is set to the current time plus 15 min. Buttons are provided to adjust the clock forward or backward in 5 or 15 min increments, with the minimum value accepted being the current time. The properties of this dialog box are that it remains in front of all other applications, is partially transparent (i.e., objects behind the dialog are viewable), and all controls on the screen can be activated while the dialog remained displayed. The dialog included a note telling the provider: "If the case takes 15 min longer than your estimate, you will be asked again" (Fig. 1). The elicited time is sent back to the database and replaces the scheduled case duration for status displays, the decision support system, and future elicited times.
Our instant messages appeared as dialogs with forced responses (Figs. 1 and 6). Based on responses to ad hoc messages sent by the OR director using the instant messaging system, we expected that availability of a free text option would sometimes result in answers about the time of patient exit that are reported not in units of time (e.g., "as soon as the x-ray is taken for the missing sponge" instead of "2:35 pm"). Thus, we do not know what would have been the perceptions of a full instant message client with two-way communication between the OR control desk and anesthesia provider, or among anesthesia providers.35
|
APPENDIX 4: ASSESSING SYSTEM "UP TIME" AND USER RESPONSIVENESS
We analyzed data from the dialogs after a 1 wk test period. Approximately 75 different anesthesiology residents and certified registered nurse anesthetists were doing cases at the studied suite during the period.
First, we estimated the fraction of time that our implementation of the Bayesian method was running. The numerator was the sum of the OR times for cases with an instant message response from the dialogs of Figure 1 and the denominator was the sum of the OR times for all cases for which an instant message should have been sent. The resulting value was equivalent to the "up-time" of an e-mail server. If the up-time were 99.0%, then 99 times out of 100 that you try to login to check your e-mail you would be able to do so. Our estimate for the up-time was, deliberately, an underestimate of the true up-time, because the automatic method did not actually fail for entire cases. Failure was not caused by characteristics of cases but by processes on the computer from which the queries were launched (e.g., system crashes and accidental interruptions of the program running the queries). No down time was generated by failures of the Bayesian program or running the associated queries. In addition, for the 20 workdays studied, the queries were being executed from a desktop computer being used concurrently for other work, whereas a production server is currently used. Statistical analysis was appropriate for the time series of failures. For each of the workdays, the number of whole minutes of cases with and without failure was calculated, and from that the Freeman-Tukey transformation of the proportion up-time.36 Student's t-distribution was used to calculate the mean and 95% CI for the 20 transformed values, and then the inverse was taken.37
Second, we analyzed the time anesthesia providers took to respond to the dialog. The interruptions caused by the dialogs are so called "negotiated" interruptions.38 This means that first the computer announces its need to interrupt the respondent, who then chooses when to deal with the interruption. Because of the special properties of the dialog, users can reply immediately, continue to work with the dialog displayed, or defer their response for 5 min at which time the dialog reappears (Fig. 1). Alternative methods of interruptions are "immediate" requiring response before regular work can proceed, "scheduled" that occur at prespecified times, and "mediated" that are held until the estimated workload of the respondent is low.38 We chose to use negotiated interruptions based on experimental findings that negotiated interruptions permitted the best performance at the respondent's primary work, among the different types of interruptions.38 Furthermore, the accuracy of the task initiated by a negotiated interruption was as good as for any of the other types of interruptions.38 Experimental participants said that they preferred the negotiated interruptions over both immediate and scheduled interruptions, partly based on their feeling less interrupted.38 However, these advantages were obtained at the expense of the longest latencies14 among the four different types of interruptions.38 Therefore, we developed,14 tested,14 and applied a method to examine percentiles of the times that the anesthesia providers took to reply to the dialogs, including the times of all deferrals. However, suppose that a provider receives the dialog and is about to close the anesthesia record on the workstation and leave the OR. There is no reason to reply.14 Thus, the times analyzed were the earlier of the times of exit from the OR as recorded in the AIMS and the time that the provider clicked the "Acknowledge" button (Fig. 1). The 95% CI for the percentiles were calculated using the conservative Clopper-Pearson method.39,40
APPENDIX 5: OBSERVATIONS AND RESULTING DESIGN DECISIONS
Appendix 1 describes the deletion of 1.0% of cases based on xsk* and xk* applied to the historical data of 128,056 cases from 2001 to 2007, not to the new data from 2008. For new cases, we considered using proactive alerts. The Bayesian method can detect whether a new case has a statistically significant (P < 0.05) chance of taking less or more time than scheduled8,26:
|
|
For example, there was a 99.8% chance that the laparoscopic Nissen fundoplication scheduled for 75 min would take longer than scheduled, based on the surgeon's average of 203 min for the previous 97 times the procedure was scheduled. However, for only 2.7% of the 129,380 cases was Eq. 12 satisfied (i.e., aberrant scheduled durations are rare). Furthermore, among those occurrences, the nk 16 cases, the 5th percentile of nk was 30 cases (95% CI 30–30), and the 10th percentile of nk was 41 cases (95% CI 35–45). Thus, as for the other hospital at which we applied Eq. 12,26 the nk sufficiently exceeded in Eqs. 2 and 4 that the xsk* had negligible influence on the resulting prediction bound of Eq. 11. Therefore, our Results are based on no alerts being used about the data entered including outliers in the scheduled duration. Instead, the system simply ignores scheduled durations of <15 min, setting xsk* = µk* for xsk* < ln(15 min).
There were n = 373 acknowledgments from the dialog of Figure 1. We calculated the median pairwise difference between: i) the absolute error between the actual time of patient exit and Eq. 11's estimate of the median time of patient exit at the time of acknowledgment and ii) the absolute error between the actual time of patient exit and the time elicited by the dialog inquiry. The median was used, because there were some (
3%) large outliers. As described in the preceding paragraph, outliers were not excluded because they do not affect the decision support system's recommendations. The absolute error, and not the proportional error, was used because the absolute error measures the cost of the forecast error (e.g., for add-on, moving case, and relief decisions2).17 The CI for the median was calculated using the Clopper-Pearson method.39,40
Since the anesthesia providers' estimates were more accurate than the Bayesian method, we continued the use of the dialogs, despite the resulting interruption. However, because the difference was only 4.7 min (1.8% of median OR time, 95% CI: 2.0–9.4 min) we limited use of the instant messages to cases exceeding the scheduled duration plus 13 min (see Appendix 3). If the studied hospital had been classifying its cases not just based on surgeon and scheduled procedure(s), but also based on type of anesthetic and surgical team,19,20,41 likely the Bayesian method would have been more accurate.
Spearman rank correlation was tested between the proportion of like cases completed at the acknowledged time (pk|dmin, Eq. 10) and the case's difference between the absolute errors. The CI for the rank correlation was calculated from the 373 paired values. There was no significant association between i) the difference in absolute errors and ii) the number of historical OR times (Spearman r = 0.09, 95% CI –0.09 to 0.27). Thus, we based use of the instant messages solely on exceedance of the scheduled duration plus 13 min (see Appendix 3), not on characteristics of the case's estimated parameters.
We used least absolute values regression to estimate the linear combination of intercept, Bayesian estimate, and elicited time remaining that minimized the absolute error in OR time remaining from the time of acknowledgment (Systat 12, simplex method). The use of least absolute values regression matches the rationale described two paragraphs above. The 95% CI of parameters were consistent with a model of the estimate being 1/3rd Bayesian + 2/3rd elicited time. There was only a 0.6 min reduction in error achieved by calculated a weighted combination of the elicited time and Bayesian estimate (0.2% of median OR time, 95% CI –0.4 to 1.5 min). Thus, we decided that each elicited time would fully replace its case's (original) scheduled OR time.
The median pairwise difference in absolute error was calculated between the regression estimate and that of elicited time alone. The estimated proportions of cases of the same combination of surgeon and procedure(s) that were expected to have finished before the times of acknowledgment were 0.47 (25th percentile), 0.65 (50th), and 0.80 (75th), respectively. There was no pairwise association between i) the proportions of like cases that would be finished by the time of acknowledgment and ii) the difference in absolute errors (Spearman r = 0.03, 95% CI –0.07 to 0.13, n = 373).
For the last 2 wk of the trial period (n = 187 cases), we tried to elicit more accurate estimates by using what we thought was a more sophisticated dialog (Fig. 6). The new display included Bayesian estimates8 for the 5th and 90th percentiles of the time of OR exit from equation. These were obtained, respectively, by setting g = 0.05 and g = 0.90 in Eq. 11. The limits of the digital clock were anchored such that the selected time could not be earlier than the 5th percentile or later than the 90th percentile, with times selectable to the nearest 5 min. The clock's default value was set equal to the g = 0.50 Bayesian estimate of the time of OR exit. We calculated the Hodges-Lehmann estimate for the median difference in the anesthesia provider's absolute errors with (Fig. 6) versus without (Fig. 1) the anchors (StatXact-7, Cytel Software Corporation, Cambridge, MA).42 There was a trend toward larger absolute errors as when compared with the use of unanchored dialogs (difference 1.3 min, 95% CI –1.6 to 4.1 min). Being a process improvement project, we promptly returned to the use of Figure 1. Perhaps future experimental studies with OR management displays21,43 could omit the default value or omit the anchors while keeping the provided information.
Footnotes
Accepted for publication October 6, 2008.
FD is the Director of the Division of Management Consulting of the Department of Anesthesia of the University of Iowa. He receives no funds personally other than his salary from the State of Iowa, including no travel expenses or honoraria, and has tenure with no incentive program.
Franklin Dexter is editor of Economics, Education, and Policy for the Journal. This manuscript was handled by Steven L. Shafer, Editor-in-chief, and Dr. Dexter was not involved in any way with the editorial process or decision.
Presented at the American Society of Anesthesiologists' meeting in Orlando, FL, on October 19, 2008. A presentation was also given at the Institute for Operations Research and the Management Sciences' meeting in Washington, DC, October 12, 2008.
REFERENCES
- Dexter F, Macario A, O'Neill L. A strategy for deciding operating room assignments for second-shift anesthetists. Anesth Analg 1999;89:920–4
[Abstract/Free Full Text] - Dexter F, Epstein RD, Traub RD, Xiao Y. Making management decisions on the day of surgery based on operating room efficiency and patient waiting times. Anesthesiology 2004; 101:1444–53[Web of Science][Medline]
- Nemeth C, O'Connor M, Klock PA, Cook R. Discovering healthcare cognition: the use of cognitive artifacts to reveal cognitive work. Organ Stud 2006;27:1011–35
- Xiao Y, Strader M, Hu P, Wasei M, Wieringa P. Visualization techniques for collaborative trajectory management. ACM Conference on Human Factors in Computing Systems 2006;1547–52
- Healey AN, Sevdalis N, Vincent CA. Measuring intra-operative interference from distraction and interruption observed in the operating theatre. Ergonomics 2006;49:589–604[Medline]
- Healey AN, Primus CP, Koutantji M. Quantifying distraction and interruption in urological surgery. Qual Saf Health Care 2007;16:135–9
[Abstract/Free Full Text] - Xiao Y, Dexter F, Hu P, Dutton RP. Usage of distributed displays of operating room video when real-time occupancy status was available. Anesth Analg 2008;106:554–60
[Abstract/Free Full Text] - Dexter F, Ledolter J. Bayesian prediction bounds and comparisons of operating room times even for procedures with few or no historical data. Anesthesiology 2005;103:1259–67[Web of Science][Medline]
- Epstein RH, Dexter F, Piotrowski E. Automated correction of room location errors in anesthesia information management systems. Anesth Analg 2008;107:965–71
[Abstract/Free Full Text] - Dexter F, Macario A. What is the relative frequency of uncommon ambulatory surgery procedures in the United States with an anesthesia provider? Anesth Analg 2000;90:1343–7
[Abstract/Free Full Text] - Dexter F, Traub RD, Fleisher LA, Rock P. What sample sizes are required for pooling surgical case durations among facilities to decrease the incidence of procedures with little historical data? Anesthesiology 2002;96:1230–6[Web of Science][Medline]
- Xiao Y, Hu P, Hao H, Ho D, Dexter F, Mackenzie CF, Seagull FJ, Dutton R. Algorithm for processing vital sign monitoring data to remotely identify operating room occupancy in real-time. Anesth Analg 2005;101:823–9
[Abstract/Free Full Text] - Sandberg WS, Sandberg EH, Seim AR, Anupama S, Ehrenfeld JM, Spring SF, Walsh JL. Real-time checking of electronic anesthesia records for documentation errors and automatically text messaging clinicians improves quality of documentation. Anesth Analg 2008;106:192–201
[Abstract/Free Full Text] - Epstein RH, Dexter F, Ehrenfeld JM, Sandberg WS. Implications of event entry latency on anesthesia information management system decision support systems. Anesth Analg 2009;108:941–7
[Abstract/Free Full Text] - Riley R, Forsyth R, Manias E, Iedema R. Whiteboards: mediating professional tensions in clinical practice. Commun Med 2007;4:165–75[Medline]
- Dexter F, Lee JD, Dow AJ, Lubarsky DA. A psychological basis for anesthesiologists' operating room managerial decision-making on the day of surgery. Anesth Analg 2007;105:430–4
[Abstract/Free Full Text] - Dexter F, Traub RD, Qian F. Comparison of statistical methods to predict the time to complete a series of surgical cases. J Clin Monit Comput 1999;15:45–51[Web of Science][Medline]
- Zhou J, Dexter F, Macario A, Lubarsky DA. Relying solely on historical surgical times to estimate accurately future surgical times is unlikely to reduce the average length of time cases finish late. J Clin Anesth 1999;11:601–5[Web of Science][Medline]
- Strum DP, Sampson AR, May JH, Vargas LG. Surgeon and type of anesthesia predict variability in surgical procedure times. Anesthesiology 2000;92:1454–67[Web of Science][Medline]
- Dexter F, Dexter EU, Masursky D, Nussmeier NA. Systematic review of general thoracic surgery articles to identify predictors of operating room case durations. Anesth Analg 2008;106:1232–41
[Abstract/Free Full Text] - Dexter F, Willemsen-Dunlap A, Lee JD. Operating room managerial decision-making on the day of surgery with and without computer recommendations and status displays. Anesth Analg 2007;105:419–29
[Abstract/Free Full Text] - Dexter F, Macario A, Traub RD. Which algorithm for scheduling add-on elective cases maximizes operating room utilization? Use of bin packing algorithms and fuzzy constraints in operating room management. Anesthesiology 1999;91:1491–500[Web of Science][Medline]
- Dexter F, Traub RD. How to schedule elective surgical cases into specific operating rooms to maximize the efficiency of use of operating room time. Anesth Analg 2002;94:933–42
[Abstract/Free Full Text] - McIntosh C, Dexter F, Epstein RH. Impact of service-specific staffing, case scheduling, turnovers, and first-case starts on anesthesia group and operating room productivity: tutorial using data from an Australian hospital. Anesth Analg 2006; 103:1499–516
[Abstract/Free Full Text] - Xiao Y, Kiesler S, Mackenzie CF, Kobayashi M, Plasters C, Seagull FJ, Fussell S. Negotiation and conflict in large scale collaboration: a preliminary field study. Cogn Tech Work 2007;9:171–6
- Dexter F, Macario A, Ledolter J. Identification of systematic under-estimation (bias) of case durations during case scheduling would not markedly reduce over-utilized operating room time. J Clin Anesth 2007;19:198–203[Medline]
- Wachtel RE, Dexter F. Simple method for deciding what time patients should be ready on the day of surgery without procedure-specific data. Anesth Analg 2007;105:127–40
[Abstract/Free Full Text] - Dexter F, Traub RD. Statistical method for predicting when patients should be ready on the day of surgery. Anesthesiology 2000;93:1107–14[Web of Science][Medline]
- Peterson C, Miller A. Mode, median, and mean as optimal strategies. J Exp Psychol 1964;68:363–7[Web of Science][Medline]
- Macario A, Dexter F. Estimating the duration of a case when the surgeon has not recently performed the procedure at the surgical suite. Anesth Analg 1999;89:1241–5
[Abstract/Free Full Text] - Press WH, Teukolsky SA, Vetterling WT, Flannery BP. Numerical recipes in FORTRAN. The Art of scientific computing. 2nd ed. New York: Cambridge University Press, 1992:207,220–222
- Johnson NL, Kotz S, Balakrishnan N. Continuous univariate distributions. Vol 2. 2nd ed. New York: John Wiley & Sons, 1995:366
- Chang DS. Critical time of the lognormal distribution. Microelectron Reliab 1994;34:261–6
- Chang DS. Optimal burn-in decision for products with an unimodal failure rate function. Eur J Oper Res 2000;126:534–40[Web of Science]
- Garrett RK, Danziger JN. IM=Interruption management? Instant messaging and disruption in the workplace. J Comput Mediat Commun 2007;13:article 2
- Mosteller F, Youtz C. Tables of the Freeman-Tukey transformations for the binomial and Poisson distributions. Biometrika 1961;48:433–40
[Free Full Text] - Dexter F, Marcon E, Epstein RH, Ledolter J. Validation of statistical methods to compare cancellation rates on the day of surgery. Anesth Analg 2005;101:465–73
[Abstract/Free Full Text] - McFarlane DC. Comparison of four primary methods for coordinating the interruption of people in human–computer interaction. Hum Comput Interact 2002;17:63–139
- Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 1934;26:404–13
[Free Full Text] - Hahn GJ, Meeker WQ. Statistical intervals. A guide for practitioners. New York: Wiley, 1991:82–4,100–5
- Combes C, Meskens N, Rivat C, Vandamme JP. Using a KDD process to forecast the duration of surgery. Int J Prod Econ 2008;112:279–93
- Maritz JS, Wu M, Stuadte RG Jr. A location estimator based on a U-statistic. Ann Stat 1977;5:779–86[Web of Science]
- Dexter F, Xiao Y, Dow AJ, Strader MM, Ho D, Wachtel RE. Coordination of appointments for anesthesia care outside of operating rooms using an enterprise-wide scheduling system. Anesth Analg 2007;105:1701–10
[Abstract/Free Full Text]
This article has been cited by other articles:
|
|
 |
|
  B. Smallman and F. Dexter Optimizing the Arrival, Waiting, and NPO Times of Children on the Day of Pediatric Endoscopy Procedures Anesth. Analg., March 1, 2010; 110(3): 879 - 887. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. S. Stepaniak, C. Heij, G. H. H. Mannaerts, M. de Quelerij, and G. de Vries Modeling Procedure and Surgical Times for Current Procedural Terminology-Anesthesia-Surgeon Combinations and Evaluation in Terms of Case-Duration Prediction and Operating Room Efficiency: A Multicenter Study Anesth. Analg., October 1, 2009; 109(4): 1232 - 1245. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
F. Dexter, R. H. Epstein, R. L. Elgart, and J. Ledolter Forecasting and Perception of Average and Latest Hours Worked by On-Call Anesthesiologists Anesth. Analg., October 1, 2009; 109(4): 1246 - 1252. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. J. Pandit and F. Dexter Lack of Sensitivity of Staffing for 8-Hour Sessions to Standard Deviation in Daily Actual Hours of Operating Room Time Used for Surgeons with Long Queues Anesth. Analg., June 1, 2009; 108(6): 1910 - 1915. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Macario Truth in Scheduling: Is It Possible to Accurately Predict How Long a Surgical Case Will Last? Anesth. Analg., March 1, 2009; 108(3): 681 - 685. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
