OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add microsoft/skills-for-fabric --skill hdinsight-migration
Ports HDInsight Spark and Hive workloads to Fabric, converting HiveContext and SparkContext to SparkSession, WASB and ABFS paths to OneLake shortcuts, Hive DDL to Delta tables in a Lakehouse, and Oozie coordinators and actions to pipeline activities and triggers. Use when the source workload is HDInsight.
npx skills add microsoft/skills-for-fabric --skill hdinsight-migration
Telemetry — MANDATORY. Every
api.fabric.microsoft.comcall must carryx-ms-fabric-skill: hdinsight-migration(az rest:--headers "x-ms-fabric-skill=hdinsight-migration"), including every LRO poll,fabric_lroand retry. Snippets omit it — add it anyway.
CRITICAL NOTES
- To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
- To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
- HDInsight has no
mssparkutilsordbutilsequivalent —notebookutilsis net-new capability being introducedHiveContextandSQLContextare legacy Spark 1.x/2.x APIs — Fabric uses Spark 3.xSparkSessionexclusivelywasb://paths are deprecated and require a Storage Account key or SAS — replace with OneLake shortcuts- Fabric cannot read or shortcut
hdfs://directly. Its Pipeline HDFS connector supports Anonymous authentication only; export or bridge Kerberos sources to supported storage first.
Read these companion documents before executing migration tasks:
az rest, az login, token acquisition, Fabric REST via CLIFor notebook and Lakehouse creation, see spark-cli. For Fabric Warehouse DDL/DML authoring, see sqldw-cli.
| Topic | Reference |
|---|---|
| Migration Workload Map | § Migration Workload Map |
| SparkSession & Context API Changes | § SparkSession API Changes |
| WASB / ABFS → OneLake Path Migration | path-migration.md |
| Hive DDL → Delta Lake / Lakehouse Schemas | hive-to-delta.md |
| Oozie → Fabric Pipelines | § Oozie → Fabric Pipelines |
Introducing notebookutils | § Introducing notebookutils |
| Before/After Code Patterns | code-patterns.md |
| Spark Configuration Differences | § Spark Configuration Differences |
| Must / Prefer / Avoid | § Must / Prefer / Avoid |
| Authentication & Token Acquisition | COMMON-CORE.md § Authentication |
| Lakehouse Management | SPARK-AUTHORING-CORE.md § Lakehouse Management |
| HDInsight Component | Fabric Target | Notes |
|---|---|---|
| Spark cluster (notebooks, scripts) | Fabric Spark (Lakehouse / Notebooks / SJD) | No persistent cluster — Starter Pool or Custom Pool provides on-demand Spark |
| Hive / HiveServer2 | Lakehouse SQL Endpoint + Lakehouse schemas | Delta Lake replaces Hive metastore; schemas provide namespace equivalent |
| HBase | Fabric Warehouse or Azure Cosmos DB (separate from Fabric) | HBase has no direct Fabric equivalent — assess workload access patterns |
| Oozie workflows | Fabric Data Pipelines | Map Oozie actions to Fabric activities; see § Oozie → Fabric Pipelines |
| YARN Resource Manager | Fabric Spark monitoring (Spark UI, Monitoring Hub) | No YARN — Fabric manages compute automatically |
| Ambari | Fabric Monitoring Hub + Admin Portal | Cluster health, capacity, and job monitoring |
| WASB / ABFS storage | OneLake Shortcuts → abfss://workspace@onelake.dfs.fabric.microsoft.com/ | See path-migration.md |
| Ranger policies | Fabric workspace roles + OneLake data access roles | Map Ranger row/column filters to Lakehouse row-level security |
| Livy REST server | Fabric Livy API | Compatible endpoint — see SPARK-AUTHORING-CORE.md |
HDInsight Spark clusters often use legacy Spark 1.x / 2.x API styles. Replace all of these with the unified SparkSession:
| Legacy HDInsight Pattern | Fabric Spark 3.x Replacement |
|---|---|
from pyspark import SparkContext; sc = SparkContext() | Not needed — sc = spark.sparkContext (pre-instantiated) |
from pyspark.sql import HiveContext; hc = HiveContext(sc) | Not needed — spark session has Hive-compatible SQL support via Delta schemas |
from pyspark.sql import SQLContext; sqlc = SQLContext(sc) | Not needed — use spark.sql(...) directly |
SparkSession.builder.enableHiveSupport().getOrCreate() | Not needed in Fabric — spark is pre-built and available |
sc.textFile("wasb://container@account.blob.core.windows.net/path") | spark.read.text("abfss://workspace@onelake.dfs.fabric.microsoft.com/lh.Lakehouse/Files/path") |
sqlContext.sql("CREATE TABLE ... STORED AS ORC") | See hive-to-delta.md for Delta DDL equivalent |
In Fabric notebooks,
spark(SparkSession) andsc(SparkContext) are pre-instantiated — do not callSparkContext()orSparkSession.builder...getOrCreate()at the top of migrated notebooks.
Map Oozie workflow actions to Fabric Data Pipeline activities:
| Oozie Action Type | Fabric Pipeline Activity | Notes |
|---|---|---|
<spark> action | Notebook activity or Spark Job Definition activity | Pass parameters via notebook cell parameters or SJD arguments |
<hive> action | Script activity (SQL) against Lakehouse SQL Endpoint | Convert HiveQL to Spark SQL or Delta SQL |
<shell> action | Azure Function activity or Web activity | Shell scripts must be refactored; no direct shell execution in Fabric Pipelines |
<java> action | Azure Batch activity (external) or refactor to PySpark | Java MapReduce jobs must be rewritten |
<sqoop> action | Copy Data activity (Fabric Data Factory connector) | Sqoop import/export maps to Fabric Copy Data with JDBC source/sink |
<coordinator> (time-based schedule) | Pipeline schedule trigger | Set recurrence in pipeline trigger; supports cron-like expressions |
<coordinator> (data-triggered) | Storage Event trigger | Trigger on OneLake file arrival |
Delegate to
spark-clifor notebook and SJD creation after mapping pipeline activities.
notebookutilsHDInsight Spark had no built-in utility framework equivalent to mssparkutils or dbutils. When migrating to Fabric, introduce notebookutils for common operations:
| Operation | Old HDInsight Approach | notebookutils Equivalent |
|---|---|---|
| List files | dbutils (N/A) / HDFS CLI | notebookutils.fs.ls("abfss://...") |
| Copy file | HDFS API / shutil | notebookutils.fs.cp(src, dest) |
| Read secret | Azure Key Vault REST call | notebookutils.credentials.getSecret(keyVaultUrl, secretName) |
| Get notebook context | Not available | notebookutils.runtime.context — returns workspace ID, notebook ID, etc. |
| Run child notebook | Not available | notebookutils.notebook.run("notebook_name", timeout, {"param": "value"}) |
| Exit notebook with value | sys.exit() | notebookutils.notebook.exit("value") |
| Mount storage | WASB config in spark-defaults.conf | OneLake Shortcut (no runtime mount needed) |
| HDInsight Concept | Fabric Spark Equivalent | Migration Action |
|---|---|---|
spark-defaults.conf (cluster-wide) | Fabric Spark Workspace Settings + Environment item | Move config properties to Environment or use %%configure in notebooks |
%%configure magic | %%configure magic — identical | No change needed |
| YARN queue / resource allocation | Fabric Spark pool node size and autoscale settings | Map queue SLAs to Custom Pool configuration |
| Ambari service configs (HDFS, YARN tuning) | Not applicable — Fabric manages infrastructure | Remove; focus on application-level Spark configs |
| HDI Spark version (e.g., Spark 2.4) | Fabric Runtime 1.3 = Spark 3.5 (latest) | Test for deprecated API removals (e.g., HiveContext, RDD-style ML) |
Conda environment / bootstrap.sh | Fabric Environment item with custom libraries | Recreate conda/pip dependencies in a Fabric Environment |
hive-site.xml (metastore connection) | Not needed — Delta Lake IS the metastore in Fabric | Remove metastore config; use Lakehouse schemas for namespace organization |
wasb:// / wasbs:// paths with OneLake abfss:// paths or OneLake Shortcuts — wasb:// requires storage account keys which are not the Fabric-preferred auth modelHiveContext, SQLContext, and standalone SparkContext() — use the pre-instantiated spark session in Fabric notebooksSTORED AS ORC, LOCATION, TBLPROPERTIES) to Delta Lake DDL — see hive-to-delta.mdnotebookutils for file system operations, secret retrieval, and child notebook orchestration where HDInsight used custom scripts or direct API callsbootstrap.sh, conda envs, and runtime %pip install patterns for production workloadshdfs:// as source-cluster-only — export or ingest into ADLS Gen2 or OneLake; do not claim that a gateway adds Kerberos support to Fabric's Anonymous-only HDFS connectorSparkContext() or HiveContext() constructors in Fabric notebooks — they conflict with the pre-instantiated spark session and will raise errorshive-site.xml or external Hive metastore configuration — Fabric's Delta Lake-backed Lakehouse IS the metastore%sh magic for file system operations in production notebooks — use notebookutils.fs.* for portability and OneLake token-based authSee code-patterns.md for full before/after examples. Key quick references:
Legacy context → Fabric pre-instantiated session
WASB path → OneLake path (after shortcut creation)
Hive DDL → Delta DDL
# HDInsight
df = spark.read.parquet("wasb://raw@myaccount.blob.core.windows.net/orders/")
# Fabric
df = spark.read.parquet("Files/raw/orders/")-- HDInsight
CREATE TABLE sales_db.fact_orders (...) STORED AS ORC LOCATION 'wasb://...';
-- Fabric
CREATE SCHEMA IF NOT EXISTS sales_db;
CREATE TABLE sales_db.fact_orders (...) USING DELTA;