~/cloudwithmani.in
← cd ..
·5 min read·author: Manikanta

Introduction to Databricks — Simple Guide

A friendly, easy-to-understand guide to Databricks: core concepts, examples, and setup tips for Windows/Linux.

#databricks#spark#data-engineering

Databricks is a managed platform for running big data workloads and machine learning on Apache Spark. In simple words: Databricks gives you a ready-made environment (clusters, notebooks, storage) so you can focus on data and code instead of managing servers.

This guide explains Databricks and shows short examples (notebook cells and a simple job), and lists easy setup steps for Windows and Linux using the Databricks CLI and Databricks Connect.

Quick overview (plain words)

  • Databricks runs Apache Spark (fast distributed data processing) and packages useful tools like notebooks, job scheduling, Delta Lake, and MLflow.
  • You write code in notebooks (Python, SQL, Scala, R) or create jobs that run scripts on clusters.
  • Databricks stores data in DBFS (Databricks File System) and integrates with cloud storage (S3, ADLS) so you can read/write large datasets.

Main components (simple descriptions)

  • Workspace: where your notebooks, dashboards, and folders live (think of it as the project workspace).
  • Notebooks: interactive documents with code cells (like Jupyter) for exploration and building pipelines.
  • Clusters: managed Spark clusters (compute + Spark) that run your notebooks and jobs.
  • Jobs: scheduled or ad-hoc runs of notebooks or scripts (repeatable pipelines).
  • Delta Lake: a storage layer on top of cloud object stores that provides ACID transactions, versioning, and fast reads/writes.
  • DBFS: Databricks File System — a filesystem abstraction for storing files in the workspace (backed by cloud storage).
  • Repos: Git-backed repositories you can sync into the workspace for version control.
  • Libraries: dependencies you install on a cluster (PyPI wheel, Maven, or uploaded egg/whl).
  • MLflow: built-in experiment tracking for machine learning (metrics, models, artifacts).
  • Unity Catalog : centralized governance for catalogs and permissions.

Simple notebook examples

Open a Python notebook and run these cells.

  1. Read a CSV from cloud storage and show rows:
# In a Databricks notebook cell (Python)
# Read CSV from S3 (example)
spark.read.csv("s3a://my-bucket/data/trades.csv", header=True, inferSchema=True).show(5)
  1. Use Spark DataFrame to compute a simple aggregation:
from pyspark.sql.functions import col, sum as _sum

df = spark.read.csv("dbfs:/data/trades.csv", header=True, inferSchema=True)
summary = df.groupBy('symbol').agg(_sum(col('amount')).alias('total_amount'))
summary.show()
  1. Write a Delta table:
df.write.format('delta').mode('overwrite').save('/mnt/delta/portfolio')
# or save as a table
spark.sql("CREATE TABLE IF NOT EXISTS portfolio USING DELTA LOCATION '/mnt/delta/portfolio'")

Simple PySpark script example (job)

Save this as jobs/portfolio_job.py in a repo and attach it to a Databricks job.

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.read.csv('s3a://my-bucket/uploads/sample_trades.csv', header=True, inferSchema=True)
# simple transform
out = df.groupBy('symbol').count()
out.show()
# write results
out.write.mode('overwrite').parquet('s3a://my-bucket/output/summary/')

Then create a Databricks Job that runs this script on a cluster on a schedule.

Setup: how to get started quickly

You can use Databricks Community Edition (free, limited) or a cloud Databricks workspace (AWS/Azure/GCP). For local development, use the Databricks CLI and Databricks Connect.

1) Create a Databricks workspace

2) Databricks CLI (Windows & Linux)

The CLI helps you upload files, run jobs, and manage clusters from your terminal.

Install and configure:

# Install pip (if needed) and the Databricks CLI
pip install databricks-cli

# Configure with a Personal Access Token (PAT)
# Create a PAT in the Databricks UI (User Settings → Access Tokens)
databricks configure --token
# then enter your workspace URL and the token when prompted

Common CLI examples:

# List clusters
databricks clusters list

# Upload a file to DBFS
databricks fs cp local_script.py dbfs:/FileStore/myapp/local_script.py

# Run a job once (if job_id exists)
databricks jobs run-now --job-id 123

3) Databricks Connect (run code from your laptop against a remote cluster)

Databricks Connect lets you run code in your local IDE while executing on a remote Databricks cluster — useful for debugging.

General steps:

  • In Databricks UI: get the cluster's spark version and download the matching databricks-connect package instructions.
  • Locally:
pip install -U databricks-connect==<matching-version>
# then configure
databricks-connect configure
# supply host, token, cluster id, org id (if required)
  • Now running spark from your local script will execute on the remote cluster.

Notes: Databricks Connect versions must match the cluster runtime; check the docs for exact matching.

Mount cloud storage (example: AWS S3)

In Databricks you can mount an S3 bucket to DBFS so you can use simple paths:

# In a notebook
dbutils.fs.mount(
  source='s3a://my-bucket',
  mount_point='/mnt/mybucket',
  extra_configs={'fs.s3a.access.key': '<AK>', 'fs.s3a.secret.key': '<SK>'}
)

For production, use instance roles or credential passthrough instead of storing keys.

Run a job via REST API (simple example)

You can trigger jobs programmatically. Example (curl):

curl -X POST https://<databricks-instance>/api/2.1/jobs/runs/submit \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "run_name": "quick-run",
    "existing_cluster_id": "<CLUSTER_ID>",
    "notebook_task": {"notebook_path": "/Repos/my/repo/notebooks/etl"}
  }'

Local development workflow (recommended)

  • Keep code in a Git repo and sync to Databricks Repos.
  • Use Databricks Connect for iterative development in your IDE.
  • Use the Databricks CLI for automation (upload files, trigger jobs).

Simple costs note (plain language)

  • Databricks charges for cluster compute (cost per node/hour) and some managed features. Stop clusters when not used to avoid charges.
  • Use small clusters for development and larger clusters for heavy jobs.

Troubleshooting quick tips

  • Notebook can't import a package? Install the library on the cluster (Libraries → Install New).
  • File not found? Verify DBFS path or S3 mount and permissions.
  • Jobs failing due to memory: use a bigger cluster or increase driver/executor memory.

Final words

Databricks is a convenient platform for Spark-based analytics and ML. Start with notebooks and small jobs, learn how Delta Lake and DBFS work, and then scale to scheduled jobs.