Verbose schema drift doc (#45054)

* this adds a verbose schema drift explanation

* weirdly missing from my previous PR

* Update doc/admin/how-to/schema-drift.md

Co-authored-by: Eric Fritz <eric@sourcegraph.com>

* clarify the safe drift case

Co-authored-by: Eric Fritz <eric@sourcegraph.com>

doc/admin/how-to/manual_database_migrations.md

@@ -41,11 +41,12 @@ upgrade \
 **Notes**:
 
 - Successive invocations of this command will re-attempt the last failed or attempted (but incomplete) migration. This command run as if the `-ignore-single-{dirty,pending}-log` flags supplied by the commands `up`, `upto`, and `downto` were enabled.
+- This command checks that the schema of the database is in the correct state for the current version, if schema drift is detected it must be resolved before completing the upgrade. [Learn more here.](./schema-drift.md).
 - Successive invocations of this command may *cause* database drift when partial progress is made. When making a subsequent upgrade attempt, invoke this command with `-skip-drift-check` ignore the failing startup check.
 
 ### drift
 
-The `drift` command describes the current (live) database schema and compares it against the expected schema at the given version. The output of this command will include all relevant schema differences that could affect application correctness and performance. When schema drift is detected, a diff of the expected and actual Postgres object definitions will be shown, along with instructions on how to manually resolve the disparity.
+The `drift` command describes the current (live) database schema and compares it against the expected schema at the given version. The output of this command will include all relevant schema differences that could affect application correctness and performance. When schema drift is detected, a diff of the expected and actual Postgres object definitions will be shown, along with instructions on how to manually resolve the disparity. [Learn more here.](./schema-drift.md)
 
 ```
 drift \
@@ -53,11 +54,10 @@ drift \
     [-version=<version>] \
     [-file=<path to description file>]
 ```
-> Note: the drig `-version` argument requires the version to be specified with the `v` prefix, for example `-version=v3.40.2`
 
 **Required arguments**:
 
-- `-db`: The target schema to inspect.
+- `-db`: The target schema to inspect. *Ex: frontend, codeintel, codeinsights*
 
 **Mutually exclusive arguments**:
 
@@ -283,7 +283,7 @@ yq eval -i \
   configure/migrator/migrator.Job.yaml
 ```
 
-The corresponding changes the `configure/migrator/migrator.Job.yaml` file would look something like this:
+The above `yq` commands alter the `configure/migrator/migrator.Job.yaml` file. For example producing the following key pairs:
 ```yaml
 image: "index.docker.io/sourcegraph/migrator:4.1.3@sha256:0dc6543f0a755e46d962ba572d501559915716fa55beb3aa644a52f081fcd57e"
 ```

doc/admin/how-to/schema-drift.md

@@ -0,0 +1,87 @@
+# How to use `migrator` operation `drift`
+
+During an upgrade you may run into the following message.
+
+```
+* Sourcegraph migrator v4.1.3
+❌ Schema drift detected for frontend
+💡 Before continuing with this operation, run the migrator's drift command and follow instructions to repair the schema. See https://docs.sourcegraph.com/admin/how-to/manual_database_migrations#drift for additional instructions.
+```
+
+This error indicates that `migrator` has detected some difference between the state of the schema in your database and the expected schema for the database in the `-from` or current version of your Sourcegraph instance.
+
+When the schema [drift](./manual_database_migrations.md#drift) command is run you'll see a set of diffs representing the areas where your instance schema has diverged from the expected state as well as the SQL operations to fix these examples of drift. For example:
+
+```
+❌ Missing index "external_service_repos"."external_service_repos_repo_id_external_service_id_unique"
+💡 Suggested action: define the index.
+
+ALTER TABLE external_service_repos ADD CONSTRAINT
+external_service_repos_repo_id_external_service_id_unique UNIQUE
+(repo_id, external_service_id);
+```
+
+```
+❌ Unexpected properties of column "batch_spec_resolution_jobs"."batch_spec_id"
+
+schemas.ColumnDescription{
+  	Name:                   "batch_spec_id",
+  	Index:                  -1,
+  	TypeName:               "integer",
+- 	IsNullable:             false,
++ 	IsNullable:             true,
+  	Default:                "",
+  	CharacterMaximumLength: 0,
+  	... // 5 identical fields
+  }
+
+💡 Suggested action: change the column nullability constraint.
+
+ALTER TABLE batch_spec_resolution_jobs ALTER COLUMN
+batch_spec_id SET NOT NULL;
+```
+
+To correct these errors in the database run the suggested SQL queries via `psql` in internal databases, or via the tools provided by your cloud database provider. 
+
+*docker example*
+```
+docker exec -it pgsql psql -U sg -c 'ALTER TABLE external_service_repos ADD CONSTRAINT external_service_repos_repo_id_external_service_id_unique UNIQUE (repo_id, external_service_id);'
+```
+*kubernetes example*
+```
+kubectl -n ns-sourcegraph exec -it pgsql -- psql -U sg -c 'ALTER TABLE external_service_repos ADD CONSTRAINT external_service_repos_repo_id_external_service_id_unique UNIQUE (repo_id, external_service_id);'
+```
+
+Then check the database again with the `drift` command and proceed with your multiversion upgrade.
+
+> Note: It is possible for the drift command to detect diffs which will not prevent upgrades. For example the following drift output picked up formating differences `\n` vs `""`:
+```
+❌ Unexpected definition of function "lsif_data_docs_search_private_delete"
+strings.Join({
+    "CREATE OR REPLACE FUNCTION
+public.lsif_data_docs_search_private_",
+    "delete()\n RETURNS trigger\n LANGUAGE plpgsql\nAS
+$function$\nBEGIN\n",
+    "UPDATE lsif_data_apidocs_num_search_results_private SET
+count =",
+-   " ",
++   "\n",
+    "count - (select count(*) from oldtbl);\nRETURN NULL;\nEND
+$functio",
+    "n$\n",
+  }, "")
+💡 Suggested action: replace the function definition.
+CREATE OR REPLACE FUNCTION
+public.lsif_data_docs_search_private_delete()
+ RETURNS trigger
+ LANGUAGE plpgsql
+AS $function$
+BEGIN
+UPDATE lsif_data_apidocs_num_search_results_private SET count =
+count - (select count(*) from oldtbl);
+RETURN NULL;
+END $function$;
+```
+
+If migrator drift suggests SQL queries which don't make sense please report to support@sourcegraph.com or open an issue in the [`sourcegraph/sourcegraph`](https://github.com/sourcegraph/sourcegraph/issues/new?assignees=&labels=&template=bug_report.md&title=) repo. You may proceed with a migrator `upgrade` command using the `-skip-drift-check=true` flag.
+