Problem

There are active tasks being executed on the Task Server, and you’d like to cancel some or all of them.

Solution

Applies to MarkLogic versions 7 or higher

xquery version "1.0-ml";

declare namespace hs="http://marklogic.com/xdmp/status/host";
declare namespace ss="http://marklogic.com/xdmp/status/server";

let $max-task-duration := xs:dayTimeDuration("PT7M")
let $min-retries := 2
(: set $user-ids to desired sequence of ids :)
let $user-ids := xdmp:get-current-userid()
let $compare-time := fn:current-dateTime() - $max-task-duration
for $host as xs:unsignedLong in xdmp:hosts()
let $task-server-id :=
  xdmp:host-status($host)//hs:task-server-id
let $requests :=
  xdmp:server-status($host, $task-server-id)//ss:request-status[
    not(xs:boolean(ss:canceled))
    (: and ss:user = 7071164300007443533 :)
    (: and ss:retry-count >= $min-retries :)
    and ss:request-text = "/some/module/uri.xqy"
    and ss:start-time < $compare-time
]
return (
  text{
    "There are currently", fn:count($requests),
    "matching requests on host", xdmp:host-name($host)
  },
  for $request in $requests
  let $request-id as xs:unsignedLong := $request/ss:request-id
  let $start as xs:dateTime := $request/ss:start-time
  return (
    text{
      $request-id, "started:", $start,
      "duration:", (fn:current-dateTime() - $start),
      if (fn:true()) then (
        xdmp:request-cancel(
          $host,
          $request/ss:server-id,
          $request-id),
        "-- cancel issued"
      )
      else ()
    }
  )
)

Discussion

Under certain circumstances, it may become necessary to cancel all the tasks on the Task Server that match a specific pattern. This may be due to human error, such as someone performing a “re-replicate all documents in domain,” or due to specific conditions occurring within a production application. Note that this recipe only cancels tasks that are actively running. Queued tasks are not affected.

The http://marklogic.com/xdmp/privileges/status privilege is required. The user running this script must also have either http://marklogic.com/xdmp/privileges/cancel-my-requests (to cancel requests running as that user) or http://marklogic.com/xdmp/privileges/cancel-any-request (to cancel any other requests).

As written, this script uses multiple filters to only show tasks that have been executing for over seven minutes, have already been retried twice, and are associated with a specific module. Other criteria that may be useful include the user that the task is running under. Also note the usage of a fixed conditional that can be toggled to perform the xdmp:request-cancel(). This allows you to test your query criteria iteratively until you are satisfied that you will only cancel the desired tasks.

Below is an example of a request record returned by xdmp:server-status. Filters can be written for any of these fields.

<request-status xmlns="http://marklogic.com/xdmp/status/server">
  <request-id>796436023172923809</request-id>
  <server-id>14409436176295478539</server-id>
  <host-id>15405276691316718307</host-id>
  <transaction-id>4091631258594104359</transaction-id>
  <canceled>false</canceled>
  <modules>11527541000394886112</modules>
  <database>17515328177061313217</database>
  <root>/</root>
  <request-kind>invoke</request-kind>
  <request-text>/</request-text>
  <update>false</update>
  <start-time>2017-03-08T15:53:22.543552Z</start-time>
  <time-limit>600</time-limit>
  <max-time-limit>3600</max-time-limit>
  <user>7071164303237443533</user>
  <trigger-depth>0</trigger-depth>
  <expanded-tree-cache-hits>0</expanded-tree-cache-hits>
  <expanded-tree-cache-misses>0</expanded-tree-cache-misses>
  <request-state>running</request-state>
  <profiling-allowed>true</profiling-allowed>
  <profiling-enabled>false</profiling-enabled>
  <debugging-allowed>true</debugging-allowed>
  <debugging-status>detached</debugging-status>
  <retry-count>0</retry-count>
</request-status>

Problem

You need to cancel requests that have queued on the Task Server, without wanting to clear the entire queue or restart the host.

Solution

Applies to MarkLogic versions 7 and higher

xquery version "1.0-ml";

declare namespace hs="http://marklogic.com/xdmp/status/host";
declare namespace ss="http://marklogic.com/xdmp/status/server";

(: set debug to fn:false() to actually cancel tasks :)
let $debug := fn:true()
let $seconds-to-run as xs:integer := 3600
let $seconds-to-pause as xs:integer := 3
(: Cancel tasks running longer than: :)
let $max-task-duration := xs:dayTimeDuration("PT7M")
let $max-iterations :=
  xs:integer($seconds-to-run div $seconds-to-pause)
let $_ := xdmp:set-request-time-limit($seconds-to-run)
for $i in (1 to $max-iterations)
return (
  if ($debug) then text{"Iteration", $i, "of", $max-iterations}
  else (),
  xdmp:invoke-function(
    function () {
      for $host as xs:unsignedLong in xdmp:hosts()
      let $task-server-id :=
        xdmp:host-status($host)//hs:task-server-id
      let $compare-time :=
        fn:current-dateTime() - $max-task-duration
      for $request in
        xdmp:server-status($host, $task-server-id)
          //ss:request-status[
            not(xs:boolean(ss:canceled))
            (: and ss:user = 7071164300007443533 :)
            (: and ss:retry-count >= 2 :)
            (: and ss:request-text = "/some/module/uri.xqy" :)
            and ss:start-time < $compare-time
          ]
      let $request-id as xs:unsignedLong :=
        $request/ss:request-id
      return (
        if ($debug) then (
          text{
            "Cancelling request:", $request-id,
            "with duration:",
            fn:current-dateTime() -
              xs:dateTime($request/ss:start-time)
          }
        ) else (),
        try {
          xdmp:request-cancel(
            $host, $task-server-id, $request-id
          )
        } catch ($e) { }
      )
    },
    <options xmlns="xdmp:eval">
      <isolation>different-transaction</isolation>
    </options>
  ),
  if ($debug) then
    text{"pausing for", $seconds-to-pause, "seconds"}
  else (),
  xdmp:sleep($seconds-to-pause * 1000)
)

Discussion

When we ask for status information about the Task Server, we can find out how many tasks are queued, along with the task IDs of the tasks that are currently running, but there’s no way to inspect the queued tasks. We can clear the queue by restarting MarkLogic, but often that’s not desirable. Also, we might only want to remove some tasks from the queue. This recipe allows us to specify which tasks to remove.

Because we can only see the running tasks, we can only cancel tasks that have started running. This also means that if there are as many good tasks running (ones that we want to continue running) as there are Task Server threads (typically 16), we won’t be able to cancel any tasks until some of the good ones finish. For example, suppose we set some selection criteria, and among the currently running tasks 12 of them do not match those criteria. That leaves just four that we can look at and try to cancel. During the next iteration of the loop, those four will have been replaced by four queued tasks (if there are some queued). If none of the 12 have finished, then we can only cancel 4 more.