Skip to content

Retrying Failed Permission Grants with On-Demand Events

When a permission apply fails for specific resources, you can re-apply the permissions for just those resources by publishing one PERMISSION_SYNC event to the Kafka topic that the MSK Ops Client consumes. The connector reconciles only the named resources against your approved policies, with no manual API call and no waiting for the next scheduled sync.

Supported connectors: Databricks Unity Catalog, Databricks SQL Analytics and AWS Lake Formation.

How a retry works

sequenceDiagram
    participant C as Your service
    participant K as Kafka topic
    participant L as MSK Ops Client (Lambda)
    participant O as Ops Server
    participant P as PolicySync connector
    participant S as Data source
    participant A as Audits (Portal, audit sink)

    C->>K: Publish PERMISSION_SYNC event (resources + requestor_id)
    K->>L: Deliver event
    L->>O: Create a scoped permission-sync task
    P->>O: Poll for the task
    P->>S: Read current grants for the named resources
    P->>P: Compare with the permissions derived from your policies
    P->>S: Apply the difference (re-apply missing grants)
    P->>A: Send ON_DEMAND audits (requestor_id, result)
  1. You publish an event to the Kafka topic. It names the resources to reconcile and carries a requestor_id you choose.
  2. The MSK Ops Client creates a scoped permission-sync task in the Ops Server for the connector that owns those resources.
  3. The connector picks up the task, reads the current grants for only the named resources, compares them with the permissions derived from your policies, and applies the difference.
  4. The result is reported as ON_DEMAND audits carrying your requestor_id, shown in the Privacera Portal and delivered to your configured audit sink.

Prerequisites

  1. The Ops Server is enabled and On-Demand Sync is configured for the connector.
  2. The MSK Ops Client is deployed with the Ops Server as its destination, and its connector configuration maps your event source to the connector.
  3. The connector sends on-demand audits. This is off by default. To turn it on, add the following to the connector's custom vars file and run the Privacera Manager update:

    YAML
    CONNECTOR_ON_DEMAND_AUDIT_ENABLE: "true"
    

    This sets the connector property policysync.on-demand.audit.enable. Without it the retry still re-applies grants, but no ON_DEMAND audit is sent, so you cannot track the result.

When to retry

Decide from the failure reported in the on-demand audit. Each outcome in the audit's extraInfo.response[] has the shape { "code", "msg", "origin", "serviceCode", "count" }, and the band of code gives one rule:

code Meaning What to do
5xxx Transient failure (throttling, timeout, a 5xx from the service, network) Re-send the event.
3xxx, 4xxx Will fail the same way again (access denied, invalid or unknown principal, invalid input) Fix the cause first, then re-send.
2xxx Success Nothing to retry.
1xxx Nothing to apply. For example, 1006 means the resource does not exist at the source (status FAILED). Check the resource name before sending again.

code is Privacera's response code, not an HTTP status. The service's own status or error code (for example 429 or REQUEST_LIMIT_EXCEEDED) is reported separately in serviceCode, so a throttled request reads "code": 5001, "serviceCode": "429". Base the retry decision on code alone.

Retry deliberately

Retry only a specific failure whose reason you understand, and cap retries per resource so a persistent failure cannot loop. Do not retry automatically on a schedule. Permission retries share the connector's on-demand processing path with other on-demand events, so unnecessary retries take capacity from it.

Publishing the event

Publish the event to the topic the MSK Ops Client consumes, authenticating with the cluster's configured mechanism (IAM or mutual TLS for Amazon MSK; mutual TLS for a self-managed Kafka cluster).

JSON
{
  "type": "PERMISSION_SYNC",
  "source": "<configured-source-name>",
  "requestInfo": {
    "requestor_id": "<unique-request-id>",
    "resources": [
      { "type": "table", "values": { "...": "connector-specific keys, see below" } }
    ]
  }
}
Field Required Meaning
type Yes PERMISSION_SYNC (any letter case) selects a permission retry. Any other value, or no type, is processed as a resource sync.
source Yes Identifies the connector the event is for. Must match a source name in the MSK Ops Client's connector configuration.
requestInfo.requestor_id Yes A unique id you choose. It is echoed on the ON_DEMAND audits so you can find the result. An event without it is rejected.
requestInfo.resources Yes The resources to reconcile, each as { "type": ..., "values": { ... } }. Several resources, of the same or different types, can be listed in one event.

Resource keys per connector

The values must give the resource's full parent chain. A resource of any other type, or with a required key missing, is dropped and creates no task.

Resource type values keys
catalog catalog
schema catalog, schema
table catalog, schema, table
volume catalog, schema, volume
function catalog, schema, function
model catalog, schema, model
external_location external_location
storage_credential storage_credential
service_credential service_credential
connection connection
metastore metastore

column is not accepted; Unity Catalog grants no privileges on a column, so reconcile its table instead. A database value is treated as schema. Names are matched case-insensitively.

JSON
{
  "type": "PERMISSION_SYNC",
  "source": "<configured-source-name>",
  "requestInfo": {
    "requestor_id": "req-uc-0001",
    "resources": [
      { "type": "table", "values": { "catalog": "main", "schema": "sales", "table": "orders" } },
      { "type": "volume", "values": { "catalog": "main", "schema": "sales", "volume": "raw_files" } }
    ]
  }
}
Resource type values keys
table database, table

Database-level retries are not supported for Databricks SQL Analytics. A database's USAGE grant is issued together with its table and database permissions rather than as a policy permission of its own, so a retry cannot detect or restore it reliably. To re-apply grants, name the tables. A database resource sent for Databricks SQL Analytics is not rejected, but its result should not be relied on.

JSON
{
  "type": "PERMISSION_SYNC",
  "source": "<configured-source-name>",
  "requestInfo": {
    "requestor_id": "req-sql-0001",
    "resources": [
      { "type": "table", "values": { "database": "sales", "table": "orders" } }
    ]
  }
}
Resource type values keys
database catalog (AWS account ID), region, database
table catalog (AWS account ID), region, database, table

When the MSK Ops Client's connector configuration sets a region for the connector, that region is used and replaces any region in the event, so each Lake Formation connector reconciles resources only in its own region.

JSON
{
  "type": "PERMISSION_SYNC",
  "source": "<configured-source-name>",
  "requestInfo": {
    "requestor_id": "req-lf-0001",
    "resources": [
      { "type": "table",
        "values": { "catalog": "111122223333", "region": "us-east-1", "database": "sales", "table": "orders" } }
    ]
  }
}

The resource types a PERMISSION_SYNC event may name are set by OPS_CONNECTOR_MSK_PERMISSION_SYNC_RESOURCE_TYPE in vars.ops-bridge.yaml (CloudFormation parameter PermissionSyncResourceType). The default accepts all the types above.

How events become tasks

The MSK Ops Client receives events from Kafka in batches. Permission-sync events received together are grouped into one task per resource type for each connector: all table resources go into one task, all schema resources into another, and so on. This always applies to permission sync and is not affected by SplitBatchByResourceType, which applies only to resource sync. A permission-sync event and a resource-sync event never share a task.

Each resource in a task keeps its own requestor_id.

What a retry touches

  • Only the resources you name. Resources you do not name are never read or changed.
  • The named resource itself, no cascade. A database, schema or catalog retry reconciles the grants on that object, not on the tables inside it. To re-drive table grants, name the tables.
  • No change when nothing differs. If the data source already matches your policies, the retry completes with no change.
  • Untracked resources are reported, not scanned. A resource the connector does not track is reported with status FAILED and code 1006. It never widens the retry.
  • Never a full scan. An event with no resources, or whose resources are all dropped, creates no task.
  • Scheduled sync is unaffected. The periodic full permission sync runs as before and eventually heals any failure on its own. The event lets you re-drive a specific resource immediately.

Tracking the result

Each retry produces audits of type ON_DEMAND, shown in the Privacera Portal and delivered to your audit sink. Each task sends one STARTED audit, then one completion audit per resource you named.

What Field Value
Event type eventType ON_DEMAND
Your request id extraInfo.input.resources[0].values.requestor_id the requestor_id you sent
Sync type extraInfo.input.syncType PERMISSION_SYNC
Loader loaderType LOAD_SERVICE_PERMISSIONS
Resource extraInfo.input.resources[0].type, .values the resource this audit is about
Status status COMPLETED or FAILED (STARTED on the task-level audit)
Outcome extraInfo.response[] code, msg, origin, serviceCode, count
Task ondemandTaskId links completion audits to their STARTED audit
  • Find your result by filtering on extraInfo.input.resources[0].values.requestor_id. Expect one completion audit per resource you named.
  • Tell retries apart from resource syncs with extraInfo.input.syncType: PERMISSION_SYNC for a retry, RESOURCE_SYNC for a resource sync.
  • No audit means the event was rejected. The MSK Ops Client drops an event before any task is created when it has no requestor_id, an unknown source, no resources, an unsupported resource type, or a missing required key. Check the event against this page and re-send it.
  • Alert on dropped events. Every drop is logged in the Lambda's CloudWatch log group with the prefix DROPPED_EVENT. We recommend a CloudWatch Logs metric filter on DROPPED_EVENT with an alarm, so a rejected event is noticed rather than waited on.

Limitation: results are reported per run, not per requestor

If a single on-demand run covers resources submitted under more than one requestor_id, the status and applied policies shown on each resource reflect the whole run rather than the individual requestor. Each entry still carries its own requestor_id, so you can locate your resources, but a resource you requested may show a failure status, or list policies, from another requestor's resources processed in the same run. This happens only when on-demand audits are enabled and events from two or more requestors, for the same resource type, are grouped into one run. To keep results isolated, send retries for one requestor_id at a time and set a per-resource retry limit.