Azure HDInsight documentation

Azure HDInsight is a managed Apache Hadoop service that lets you run Apache Spark, Apache Hive, Apache Kafka, Apache HBase, and more in the cloud.

• Azure HDInsight documentation
• HDInsight service
  • Overview
    • What is Azure HDInsight?
    • Before you start
  • Tutorials
    • Create HDInsight clusters
    • Manage HDInsight clusters - Portal
    • Manage HDInsight clusters - CLI
    • Manage HDInsight clusters - .NET SDK
    • Manage HDInsight clusters - PowerShell
    • Create clusters with Runbooks
    • Apache Ambari email notifications
    • Build an ETL pipeline
  • Samples
    • Azure CLI samples
    • .NET samples
    • Java samples
    • Python samples
    • Azure samples browser
  • Concepts
    • Versioning
      • Versioning introduction
      • HDInsight versions
      • HDInsight version 4.0
      • HDInsight version 5.x
      • Retired HDInsight versions
    • Enterprise Security Package
    • High availability components
    • Apache Ambari in Azure HDInsight
    • Streaming at scale in HDInsight
    • Apache Hadoop architecture in HDInsight
    • HDInsight supported VM sizes
    • Select the right VM size
    • Cluster capacity planning
    • Cluster management best practices
    • Reliability and Business continuity
      • Reliability in HDInsight
      • Business continuity architecture
      • High availability case study
    • Understand managed identities
    • MSI Support to access Azure services
    • Compare storage options
      • Azure Storage overview
      • Azure Data Lake Storage Gen2 overview
  • How-to guides
    • Create clusters
      • Create clusters in the portal
      • Create clusters using templates
      • Create clusters using Azure CLI
      • Create clusters using PowerShell
      • Create clusters using REST API
      • Subscribe to HDInsight release notes GitHub repo
      • View the image version of Azure HDInsight cluster
      • Cluster creation errors
    • Use cluster storage
      • Use Azure Storage
      • Use Data Lake Storage Gen2
      • Use Data Lake Storage Gen2 in the portal
      • Use Data Lake Storage Gen2 with Azure CLI
      • Configure blob storage as primary storage in Azure HDInsight
    • Extend clusters
      • Install HDInsight apps
      • Customize clusters using Bootstrap
      • Customize clusters using Script Action
      • Develop script actions
      • Open-source software support
      • Add Apache Hive libraries
      • Use Hue
      • Build HDInsight applications
        • Install custom apps
        • Use REST to install apps
        • Publish HDInsight apps to Azure Marketplace
    • Secure
      • Use SSH with HDInsight
      • Use SSH tunneling
      • Use SCP with HDInsight
      • Restrict access to data
      • Authorize users for Apache Ambari Views
    • Migrate
      • Migrate .NET to Linux
      • Migrate to granular role-based access for cluster configurations
      • Ambari user configs migration
      • Upgrade to Apache Ranger in Azure HDInsight
      • Load balancer migration guidelines for Azure HDInsight
      • Hadoop
        • Motivation and benefits
        • Best practices - architecture
        • Best practices - infrastructure
        • Best practices - storage
        • Best practices - data migration
        • Best practices - security and DevOps
      • Kafka
        • Migrate Apache Kafka workloads to Azure HDInsight 5.1
        • Migrate Kafka workloads to 4.0
      • HBase
        • Migrate Apache HBase cluster to newer version
        • Migrate an Apache HBase cluster to a HDInsight 5.1
        • Migrate Apache HBase version and storage account
        • Migrate Apache HBase to a HDInsight 5.1 and new storage account
      • Interactive Query
        • Migrate default Hive metastore to external metastore on Azure HDInsight
        • Copy Hive tables across Storage Accounts
    • Manage
      • Manage clusters using the Apache Ambari web UI
      • Disable auto logout from Ambari Web UI
      • Optimize with Apache Ambari
        • Optimize clusters
        • Optimize Apache HBase
        • Optimize Apache Hive
        • Optimize Apache Pig
      • Find host names of cluster nodes
      • Reboot cluster nodes
      • Use Apache Ambari REST API
      • Delete a cluster
      • Manually scale a cluster
      • Autoscale
        • Autoscale clusters
        • Custom configure Autoscale
      • Use external metadata stores
      • Custom Ambari DB
      • Manage logs for a HDInsight cluster
      • Add storage accounts
      • Update storage account access key
      • Upload data for Apache Hadoop jobs
      • Multiple HDInsight clusters with Data Lake Storage
      • Import and export data with Apache Sqoop
        • Overview - using Apache Sqoop
        • Connect with SSH
        • Run using cURL
        • Run using SDK for .NET
        • Run using Azure PowerShell
      • Operationalize a data analytics pipeline
      • Use Apache Oozie for workflows
      • Cluster and service ports and URIs
      • Upgrade HDInsight cluster to newer version
      • OS patching for HDInsight cluster
    • Use HDInsight tools
      • Tools for Visual Studio
      • Spark and Hive Tools for VS Code
      • HDInsight Configuration Settings Reference
    • Monitoring
      • Monitor Azure HDInsight
      • Use Azure Monitor Agent
        • Azure Monitor Agent Migration
        • Selective Logging
      • Use Azure Monitor logs
        • New Azure Monitor experience Migration
        • Selective Logging
        • Enable new Azure Monitor integration
      • Use queries with Azure Monitor logs
      • Monitor cluster performance
      • Cluster availability - Apache Ambari
      • Cluster availability - Azure Monitor logs
      • Monitoring data reference
    • Troubleshoot
      • Known issues
        • Known issues in HDInsight
        • Conda version regression in a recent HDInsight release
        • Cluster Reliability Issues
        • Component version validation error in ARM templates
        • Switch Users in Ambari UI
      • Request increase for CPU core quota
      • Troubleshoot Apache Ambari Metrics Collector
      • Troubleshoot script actions
      • Resource creation failures
      • Troubleshoot Apache Oozie
      • Troubleshoot Apache Sqoop
      • Understand and resolve WebHCat errors
  • Reference
    • Azure CLI
    • Azure PowerShell
    • SDK for .NET
    • SDK for Python
    • SDK for Java
    • SDK for Go
    • .NET Apache HBase
    • REST API
    • REST API Apache Spark
    • Resource Manager template
    • Azure Policy built-ins
  • Resources
    • Releases
      • Release notes
      • HDInsight retirements and action required
      • Archived release notes for Azure HDInsight
    • Frequently asked questions FAQ
    • Information about using HDInsight on Linux
    • Apache Hadoop memory and performance
    • Access Apache Hadoop YARN application logs on Linux
    • Enable heap dumps for Apache Hadoop services
    • Get help on the Microsoft Q&A question page
    • Preview features
    • Pricing calculator
    • Windows tools for HDInsight
• Apache Spark
  • Overview
    • What is Apache Spark in HDInsight?
  • Quickstarts
    • Create Apache Spark cluster - Portal
    • Create Apache Spark cluster - PowerShell
    • Create Apache Spark cluster - Azure CLI
    • Create Apache Spark cluster - Bicep
    • Create Apache Spark cluster - ARM Template
  • Tutorials
    • Run queries on an Apache Spark cluster
    • Failure spark job debugging in IntelliJ preview
    • Analyze data using BI tools
    • Create a machine learning app
    • Create an Apache Spark app in IntelliJ
  • Concepts
    • Spark best practices on HDInsight
    • Configure Apache Spark settings
    • Optimization
      • Optimize Apache Spark jobs
      • Optimize data processing
      • Optimize data storage
      • Optimize memory usage
      • Optimize cluster configuration
  • How-to guides
    • Use tools
      • Tools for IntelliJ
      • Tools for Eclipse
      • PySpark for VS Code
    • Develop
      • Optimize queries with SparkCruise
      • Use an interactive Apache Spark Shell
      • Remote jobs with Apache Livy
      • Debug Apache Spark jobs remotely with IntelliJ through VPN
      • Apache Spark streaming
        • Streaming overview
        • Structured Streaming overview
        • Create highly available Apache Spark streaming jobs in Apache Hadoop YARN
        • Create Apache Spark streaming jobs with exactly once event processing
      • Apache Spark and Machine Learning
        • Predict food inspection results
        • Analyze website logs
        • Use with Microsoft Cognitive Toolkit
        • Create an Apache Spark machine learning pipeline
      • Analyze big data
        • Use Apache Spark to read and write Apache HBase data
        • Analyze Application Insights telemetry logs
    • Manage
      • Manage dependencies
        • Manage Spark application dependencies
        • Safely manage jar dependencies
        • Safely manage Python environment
      • Manage cluster resources
      • Manage Apache Spark applications using extended History Server
      • Enable caching with IO Cache
      • Use notebooks with Apache Spark
        • Use a local Jupyter Notebook
        • Jupyter Notebook kernels
        • Use external packages with Jupyter using cell magic
        • Use Apache Zeppelin notebooks
      • Use with other Azure services
        • Connect to Azure SQL Database
        • Run Azure Machine Learning workloads with AutoML
    • Troubleshoot
      • Can't create Jupyter Notebook
      • OutOfMemoryError exception
      • Apache Spark job fails - NoClassDefFoundError
      • Apache Spark job fails - InvalidClassException
      • Slow Apache Spark jobs - storage container
      • IllegalArgumentException exception
      • Apache Spark Streaming application stops
      • RpcTimeoutException exception
      • Blocking Cross Origin API
      • Kryo serialization failed
      • Apache Spark event log - RequestBodyTooLarge
      • Debug Apache Spark jobs
      • Debug WASB file operations
      • Use IntelliJ to debug Apache Spark job
      • Apache Spark troubleshooting
      • Known issues
• Apache Hadoop
  • Overview
    • What is Apache Hadoop in HDInsight?
  • Quickstarts
    • Create Apache Hadoop cluster - Portal
    • Create Apache Hadoop cluster - Bicep
    • Create Apache Hadoop cluster - ARM Template
  • Tutorials
    • Create on-demand clusters
    • Submit Apache Hadoop jobs
  • How-to guides
    • Use tools
      • Use Apache Pig
    • Develop
      • Use MapReduce with Apache Hadoop
        • Use MapReduce
        • Use SSH
        • Use cURL
        • Use Azure PowerShell
        • Use SDK for .NET
        • Run the MapReduce samples
        • Run custom Apache Hadoop MapReduce programs
        • Develop C# streaming MapReduce programs
        • Develop Java MapReduce programs
      • Use Apache Hive as an extract, transform, and load ETL tool
      • Extract, transform, and load at scale
      • Create non-interactive authentication .NET HDInsight applications
      • Analyze big data
        • Analyze using Power Query
        • Connect Power BI to Interactive Query
        • Connect Power BI to Apache Hadoop
        • Connect Excel to Apache Hadoop
        • Connect using the Apache Hive JDBC driver
        • Analyze stored tweets
        • Generate recommendations with Apache Mahout
        • Advanced analytics for HDInsight
      • Manage
        • Use empty edge nodes
        • Use edge nodes to scale HiveServer2
    • Troubleshoot
      • Common issues
      • Cluster creation failures
      • Out of disk space
      • Soft lockup - CPU
      • InvalidNetworkConfigurationErrorCode - cluster creation fails
      • InvalidNetworkSecurityGroupSecurityRules - cluster creation fails
      • Unable to log into HDInsight cluster
      • Unable to add nodes to HDInsight cluster
      • Converting service principal certificate
      • Local HDFS stuck in safe mode
      • Apache Ambari heartbeat issues
      • Apache Ambari UI 502 error
      • Apache Ambari shows down hosts and services
      • Apache Ambari stale alerts
      • Apache Ambari directory alerts
      • Troubleshoot a slow or failing HDInsight cluster
      • Apache Hadoop HDFS troubleshooting
      • Apache Hadoop YARN troubleshooting
      • Invalid BCFile error from Yarn log
      • Troubleshoot Data Lake Store files
      • Port conflict when starting services
      • Lost Key Vault access
      • WASBS storage exception
      • Manage disk space
• Apache Kafka
  • Overview
    • What is Apache Kafka in HDInsight?
  • Quickstarts
    • Create Apache Kafka cluster - Portal
    • Create Apache Kafka cluster - PowerShell
    • Create Apache Kafka cluster - Bicep
    • Create Apache Kafka cluster - ARM Template
  • Tutorials
    • Create Apache Kafka cluster - Azure CLI
    • Use Apache Kafka Producer and Consumer API
    • Develop app with Apache Kafka Streams API
    • Use Apache Spark structured streaming
    • Connect Kafka cluster with VM in different VNet
  • Concepts
    • Storage and scalability
    • High availability of data
    • Performance tuning
    • Kafka FAQ
  • How-to guides
    • Develop
      • Use Kafka REST proxy
      • Enable auto topic creation
      • Use Apache Kafka with IoT Hub
      • Create streaming pipelines
        • Apache Spark, Apache Kafka, and Azure Cosmos DB
        • Apache Kafka and Apache Spark DStream
        • Use Apache Kafka with Azure Kubernetes Service
    • Manage
      • Use Apache Kafka in a virtual network
      • Replicate Apache Kafka data
      • Use MirrorMaker2 to replicate Apache Kafka topics with Kafka
      • Analyze Apache Kafka logs
      • Secure Spark and Kafka streaming integration scenario
      • SSL Encryption and Authentication for Non ESP Kafka cluster
      • SSL Encryption and Authentication for ESP Kafka cluster
      • Kafka MirrorMaker 2.0 guide
      • Connect HDInsight Kafka cluster with client VM in different VNet
    • Troubleshoot
      • Insufficient fault domains
      • Full data disk
• Apache HBase
  • Overview
    • What is Apache HBase in HDInsight?
  • Quickstarts
    • Create Apache HBase cluster - Bicep
    • Create Apache HBase cluster - ARM Template
    • Query Apache HBase - Apache Phoenix
    • Query Apache HBase - HBase Shell
  • Tutorials
    • Use Apache HBase in HDInsight
  • Concepts
    • Optimize for advisor recommendations
  • How-to guides
    • Develop
      • Use the Apache HBase SDK for .NET
      • Develop Java applications
      • Apache Phoenix with Apache Zeppelin
      • Apache Phoenix in HDInsight
      • Use the Apache Phoenix Query Server REST SDK
      • Bulk load data into Apache Phoenix using psql
    • Manage
      • Accelerated Writes
      • Apache HBase cluster replication in Azure VNet
      • Set up backup and replication for Apache HBase and Apache Phoenix
      • Apache Phoenix performance best practices
      • Create Apache HBase cluster in Azure VNet
      • Use Apache HBase HBCK2 Tool
    • Troubleshoot
      • Troubleshoot Apache HBase performance issues
      • hbase hbck returns inconsistencies
      • Storage exception after connection reset
      • No data in Apache Phoenix views
      • BindException - Address in use
      • Apache HBase fails to start
      • Issues with region servers
      • Apache Phoenix connectivity issues
      • Apache HBase REST not responding
      • Pegged CPU on region server
      • Timeouts with 'hbase hbck' command
      • REST API to query Apache HBase
      • Troubleshoot data retention issues
• Interactive Query
  • Overview
    • What is Interactive Query in HDInsight?
  • Quickstarts
    • Create Interactive Query cluster - Bicep
    • Create Interactive Query cluster - ARM Template
    • Execute Apache Hive queries - Apache Zeppelin
    • Autoscaling Interactive Query clusters
  • Tutorials
    • Analyze data with Interactive Query
  • Concepts
    • Gateway best practices
    • Sizing guidelines
    • Hive replication
  • How-to guides
    • Develop
      • Process and analyze JSON documents
      • Use C# user-defined functions
      • Use Python with Apache Hive and Apache Pig
      • HWC integration with Apache Spark and Apache Hive
      • HWC and Apache Spark operations
      • HWC integration with Apache Zeppelin
      • HWC 1.0 Supported APIs
      • HWC 2.0 Supported APIs
      • Apache Hive with Hadoop
      • Use the Apache Hive View
      • Connect to Apache Beeline
      • Use Apache Hive Beeline
      • Use Grafana
      • Use REST API
      • Use Azure PowerShell
      • Use SDK for .NET
      • Use the HDInsight tools for Visual Studio
      • Use a Java UDF with Apache Hive
      • Query Hive with PowerShell and ODBC
      • Enable Hive LLAP Workload Management feature
    • Manage
      • Optimize Apache Hive queries
    • Troubleshoot
      • Apache Hive settings fix Out of Memory error
      • Apache Tez application hangs
      • Apache Hive LLAP sizing guidelines
      • Apache Hive LLAP query performance
      • Slow reducer
      • Apache Hive gateway timeout
      • Apache Hive View time-out
      • Permission error creating table
      • GC overhead limit exceeded
      • Apache Ambari Tez View loads slowly
      • Error message not shown in Apache Hive View
      • Inaccessible Apache Hive View
      • ZooKeeperHiveClientException - HiveServer2 configs
      • Apache HIVE troubleshooting
      • Apache Hive logs taking up entire disk space on Head node
      • Security options for Hive
      • Apache Hive LLAP Workload Management
  • Reference
    • Hive LLAP Workload Management commands
• Enterprise readiness
  • Overview
    • What is enterprise security in HDInsight?
  • Tutorials
    • Apache HBase RBAC with Apache Ranger
    • Apache Hive RBAC with Apache Ranger
    • Apache Kafka RBAC with Apache Ranger
    • Use Apache Oozie in ESP clusters
    • Use Apache Spark-SQL RBAC with Apache Ranger
  • Concepts
    • Security baseline
    • Enterprise security general guidelines
    • Plan for ESP clusters
    • HDInsight virtual network architecture
    • Restrict public connectivity
    • Enable private link on a HDInsight cluster
    • Enable private link on a HDInsight Kafka Rest Proxy cluster
    • Transport layer security
    • Plan VNETs for HDInsight
    • Control network traffic
    • Required IP Addresses for NSGs and UDRs
    • Customer-managed key disk encryption
    • LDAP sync in Ranger and Apache Ambari
    • Double encryption in transit
    • Use availability zones
  • How-to guides
    • Use ID Broker for credential management
    • Create ESP clusters and sync with on-premises
    • Create VNETs for HDInsight
    • Connect HDInsight with on-premises network
    • Configure ESP clusters using Microsoft Entra Domain Services
    • Synchronize Microsoft Entra users to a HDInsight cluster
    • Manage clusters with enterprise security
    • Manage SSH access
    • Securing data
    • Use firewall to restrict outbound traffic
    • Create service endpoint policies
    • Configure network virtual appliance
    • Service tags for Azure firewall
    • Troubleshoot
      • ESP Authentication issues
      • Cluster creation fails with DomainNotFound
      • Sample script to use when Cluster creation fails with DomainNotFound Error
• Azure Synapse integration
  • Share Hive metastore with Synapse Spark