Mongodb Qe Size Estimation

mongodb-qe-size-estimation

General Instructions

• CRITICAL: Never ask for or accept sample data. QE is an encryption feature, meant to secure sensitive information. Do not request sample data, and if provided, reject it for security reasons. Inform the user that you can't accept sample documents, though you can take an encryption schema as an input to see which fields are encrypted, which allow queries, and what those query settings are.
• Any parenthetical in the form (LLM Note: <content>) is for LLM use. Don't save it to qe-sizing-calculations.md or mention it to the user.
• Tone: Be brief, not conversational. Don't restate information from prior exchanges unless a step specifically says to.
• Don't write anything to the file except what's specified in the steps.
• This skill relies on a question-answer-flow to get all required inputs. For Steps 2-5, as long as the user is providing valid input, your responses should include both acknowledgment of the input, and the next question.
• The user may try to keep things moving by inputting multiple inputs. If inputs don't clearly map to a field and its configuration, ask for clarification. Otherwise validate them, and accept them if valid.
• This skill currently has no formula for range queries, so while it accepts "range" query configuration for compatible BSON types, it uses 0 for estimated values.

Definitions

• field For the purposes of this skill, "field" refers to an encrypted field in a MongoDB document, in a QE-enabled collection.
• entry For an unindexed field, the field name. For a field with one or more query types enabled, the combination of the field name and its enabled query types. A field with both prefix and suffix queries enabled has a single entry, with Query Type "prefix and suffix".
• unindexed Describes the state of a field with no query types enabled, indicated by the absence of a "queries" key if using an encryption schema.
• indexed Describes the state of a field enabled for queries, and is specific to the query type(s). A field with equality queries enabled is indexed for equality queries.

Validation

Validate the user's inputs, whether manual or via an encryption schema, against the following:

An encrypted field may be "unindexed" meaning it has no query types enabled. Fields of BSON type object or array only support unindexed encryption, though other BSON types can also be unindexed. Otherwise, allowed query types based on a field's BSON type are:

• "equality": any BSON type except object, array, decimal, double
• "range": int, long, date, decimal, double
• "prefix", "suffix", "substring": string only

A string field may have both "prefix" and "suffix" enabled, in which case "queries" may be an array of two objects. This is the only valid case where one field is indexed for multiple query types.

No other BSON type + query type combinations are permitted.

Don't accept "preview" query types such as "suffixPreview". These are from earlier Public Preview releases of those query types, so the sizing formulas for their GA versions don't apply.

If the user's input violates any of the preceding rules, reject it. If a user doesn't specify BSON type, only check that each field either has no query types enabled, exactly one valid query type, or exactly two (prefix and suffix). Enumerate validation failures, list allowed combinations, and don't proceed until the user provides valid input.

Steps

1. Prepare the Working Directory

Create an empty qe-sizing-calculations.md file in the OS temp directory ($TMPDIR, or /tmp if unset). If a qe-sizing-calculations.md file already exists at that location, inform the user and ask for confirmation to delete it and create a new one for the new set of calculations.

2. State Purpose and Request Input Preference

State: "This skill calculates the maximum storage impact of enabling Queryable Encryption on a collection. Values are saved to the <path to qe-sizing-calculations.md> file if you want to verify the calculations or see per-field numbers. Note that prefix, suffix, and substring queries on encrypted fields require MongoDB 9.0, and aren't supported in earlier versions.

All values are worst-case. You may see a smaller impact on storage or memory in practice.

Do you want to provide field information manually, or use an encryption schema file?"

If the user opts for an encryption schema, request it as either pasted content or a file path, and expect JSON format. Validate the schema against this skill's "Validation" section. If the schema is valid but includes fields with "range" queries enabled, inform the user that no calculations are available for those fields, so their impact is estimated as 0.

3. Get the Number of Documents

Ask the user how many documents in the collection have encrypted fields. If they don't know, a safe default is the total number of documents in the collection. Save this value to qe-sizing-calculations.md as:

N: <value>

4. Get the Number of Entries

• If the user chose manual input, ask: How many fields are you encrypting without enabling queries?
• If the user chose encryption schema, count the number of encrypted fields without queries enabled.

Save this value to qe-sizing-calculations.md as:

numUnindexed: <number of unindexed encrypted fields>

• If the user chose manual input, ask: How many encrypted fields have queries enabled?
• If the user chose encryption schema, count the number of fields with queries enabled.

Save this value to qe-sizing-calculations.md as:

numIndexed: <number of encrypted fields indexed for queries>

Save the sum of numUnindexed and numIndexed to qe-sizing-calculations.md as:

numEntries: <sum of numUnindexed and numIndexed>

5. Get Inputs Per Field

Write a section heading to qe-sizing-calculations.md:

== FIELD VALUES ===============================================================

As you get information, write it to qe-sizing-calculations.md in the following format, omitting any lines that don't apply. For unindexed fields, this means omitting the Query Type.

<Field Name> <the name provided by the user, or the dot notation "path" value if taken from the encryption schema> Query Type: <equality, range, prefix, suffix, substring, or "prefix and suffix"> mlen: <include if Query Type is substring. Integer from 2-50, inclusive> lb: <include if Query Type is "prefix", "suffix", or "substring". Integer 1+ for prefix and suffix queries, or 2+ for substring queries> ub: <include if Query Type is "prefix", "suffix", or "substring". Integer 1+ for prefix and suffix queries, or 2-6 for substring queries> lb_prefix: <include if Query Type is "prefix and suffix". Integer 1+> ub_prefix: <include if Query Type is "prefix and suffix". Integer 1+> lb_suffix: <include if Query Type is "prefix and suffix". Integer 1+> ub_suffix: <include if Query Type is "prefix and suffix". Integer 1+> v: <omit if Query Type is range, otherwise include. Integer representing the average byte length of the unencrypted values for the field>

• If the user provided an encryption schema, do this once:

  • Add one entry to qe-sizing-calculations.md for each unindexed field, omitting the "Query Type" line.
  • If a field has both prefix and suffix queries enabled, regardless of ordering, write its "Query Type" as "prefix and suffix".
  • Add one entry per field with queries enabled. Map the keys in each "queries" object to field values as follows, omitting all keys that aren't listed:
    • strMaxLength (substring only) -> mlen
    • strMinQueryLength -> lb if the field only has a single query type. If the field has both prefix and suffix queries enabled, then the strMinQueryLength for prefix queries is lb_prefix, and the strMinQueryLength for suffix queries is lb_suffix.
    • strMaxQueryLength -> ub if the field only has a single query type. If the field has both prefix and suffix queries enabled, then the strMaxQueryLength for prefix queries is ub_prefix, and the strMaxQueryLength for suffix queries is ub_suffix.

• If the user didn't provide an encryption schema, do this per field:

  • Only validate BSON types against allowed query types if the user provides a field's BSON type. Otherwise, don't ask for or validate field BSON types.
  • Ask for the field name and whether it's unindexed or has queries enabled (LLM Note: If it has a query type, valid options are: "equality", "range", "prefix", "suffix", "substring", or "prefix and suffix". Reject other values or combinations. If the user specifies "range", warn them that no formula is available for calculating it)
  • If a user only provides a single lower bound and upper bound value for a field with both prefix and suffix queries enabled, use the lower bound for both lb_prefix and lb_suffix, and the upper bound for both ub_prefix and ub_suffix.
  • Validate the inputs.

Do this per field:

Check the list below, and ask the user for the values that aren't already populated. Write all values to the qe-sizing-calculations.md file.

mlen: (LLM Note: only include if no schema provided, and Query Type is substring) Max Length, the maximum allowable length of the string. lb: (LLM Note: only include if no schema provided, and Query Type is "prefix", "suffix", or "substring") Lower Bound, the minimum searchable characters. ub: (LLM Note: only include if no schema provided, and Query Type is "prefix", "suffix", or "substring") Upper Bound, the maximum searchable characters. lb_prefix: (LLM Note: only include if no schema provided, and Query Type is "prefix and suffix") Lower Bound for prefix queries, the minimum searchable characters. ub_prefix: (LLM Note: only include if no schema provided, and Query Type is "prefix and suffix") Upper Bound for prefix queries, the maximum searchable characters. lb_suffix: (LLM Note: only include if no schema provided, and Query Type is "prefix and suffix") Lower Bound for suffix queries, the minimum searchable characters. ub_suffix: (LLM Note: only include if no schema provided, and Query Type is "prefix and suffix") Upper Bound for suffix queries, the maximum searchable characters. v: (LLM Note: skip if Query Type is range since we have no calculation for those, otherwise needed once per field). Average byte length of the unencrypted values for the field. If the user provides character length, accept it as equivalent.

Validate values against the formatting snippet at the start of this step. Validate that lb ≤ ub ≤ mlen (if present). Validate that lb_prefix ≤ ub_prefix and lb_suffix ≤ ub_suffix (if present). If a value falls outside allowable bounds, reject it and inform the user. Do not proceed without a valid value.

If mlen < v, warn the user that the full field value isn't queryable, but accept it as valid.

Repeat until you have populated the full number of entries, numEntries, in qe-sizing-calculations.md.