What is the difference between AWS glue vs lambda?

 In the context of AWS, here is the difference given between both:-

AWS glue

You can use the AWS glue when you are dealing with complex ETL workflow which requires automated schema discovery, data cataloging, and orchestrating.

The AWS glue is very suitable for large-scale data processing tasks where data transformation includes multiple steps and data sources.

Here is the example given in the form of coding for the AWS glue ETL job:-

From awsglue.context import GlueContext

From pyspark.sql import SparkSession

# Initialize Spark session and Glue context

Spark = SparkSession.builder.appName(“GlueETLJob”).getOrCreate()

glueContext = GlueContext(spark)

# Define source and target data paths

Source_path = “s3://my-source-bucket/input-data/”

Target_path = “s3://my-target-bucket/output-data/”

# Read data from source

Source_df = glueContext.create_dynamic_frame.from_options(

    Connection_type=”s3”,

    Connection_options={“paths”: [source_path]},

    Format=”json”

).toDF()

# Perform data transformations

Transformed_df = source_df.withColumn(“new_column”, source_df[“existing_column”] + 1)

# Write transformed data to target

Transformed_df.write.mode(“overwrite”).format(“parquet”).save(target_path)

AWS Lambda

You can opt-out of the AWS lambda for lightweight and event-driven processing tasks which would require quick implementation and scalability based on the demand. You can use lambda for processing individual records, triggering action based on the events, and integrating with the other AWS services seamlessly.

Here is the example given in the form of coding for the AWS lambda function in Python:-

Import boto3

Def lambda_handler(event, context):

    # Extract data from the event

    Input_data = event[“Records”][0][“s3”][“object”][“key”]

    # Process data (e.g., perform calculations, format data)

    Processed_data = process_data(input_data)

    # Save processed data to the target location

    Save_to_s3(processed_data, “my-target-bucket/output-data/processed-data.txt”)

Def process_data(input_data):

    # Placeholder for data processing logic

    Processed_data = input_data.upper()  # Example: Convert data to uppercase

    Return processed_data

Def save_to_s3(data, target_path):

    S3 = boto3.client(“s3”)

    S3.put_object(Body=data, Bucket=”my-target-bucket”, Key=target_path)

Here is the combination of the example given which would integrate both AWS glue and AWS lambda for a data processing pipeline:-

From pyspark.context import SparkContext

From awsglue.context import GlueContext

From pyspark.sql import SparkSession

# Initialize Spark context and Glue context

Sc = SparkContext()

glueContext = GlueContext(sc)

spark = glueContext.spark_session

# Define source and target data paths

Source_path = “s3://my-source-bucket/input-data/”

Target_path = “s3://my-target-bucket/output-data/”

# Read data from source

Source_df = glueContext.create_dynamic_frame.from_options(

    Connection_type=”s3”,

    Connection_options={“paths”: [source_path]},

    Format=”json”

).toDF()

# Perform data transformations

Transformed_df = source_df.withColumn(“new_column”, source_df[“existing_column”] + 1)

# Write transformed data to target

Transformed_df.write.mode(“overwrite”).format(“parquet”).save(target_path)