Associate-Developer-Apache-Spark問題集で2022年最新のDatabricks Associate-Developer-Apache-Spark試験問題 [Q40-Q63]

新問題 40
The code block displayed below contains an error. The code block should arrange the rows of DataFrame transactionsDf using information from two columns in an ordered fashion, arranging first by column value, showing smaller numbers at the top and greater numbers at the bottom, and then by column predError, for which all values should be arranged in the inverse way of the order of items in column value. Find the error.
Code block:
transactionsDf.orderBy(‘value’, asc_nulls_first(col(‘predError’)))

Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.

Column value should be wrapped by the col() operator.

Column predError should be sorted in a descending way, putting nulls last.

Column predError should be sorted by desc_nulls_first() instead.

Instead of orderBy, sort should be used.

新問題 41
The code block shown below should return a column that indicates through boolean variables whether rows in DataFrame transactionsDf have values greater or equal to 20 and smaller or equal to
30 in column storeId and have the value 2 in column productId. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__((__2__.__3__) __4__ (__5__))

1. select
2. col(“storeId”)
3. between(20, 30)
4. and
5. col(“productId”)==2

1. where
2. col(“storeId”)
3. geq(20).leq(30)
4. &
5. col(“productId”)==2

1. select
2. “storeId”
3. between(20, 30)
4. &&
5. col(“productId”)==2

1. select
2. col(“storeId”)
3. between(20, 30)
4. &&
5. col(“productId”)=2

1. select
2. col(“storeId”)
3. between(20, 30)
4. &
5. col(“productId”)==2

新問題 42
Which of the following describes the characteristics of accumulators?

Accumulators are used to pass around lookup tables across the cluster.

All accumulators used in a Spark application are listed in the Spark UI.

Accumulators can be instantiated directly via the accumulator(n) method of the pyspark.RDD module.

Accumulators are immutable.

If an action including an accumulator fails during execution and Spark manages to restart the action and complete it successfully, only the successful attempt will be counted in the accumulator.

新問題 43
Which of the following is not a feature of Adaptive Query Execution?

Replace a sort merge join with a broadcast join, where appropriate.

Coalesce partitions to accelerate data processing.

Split skewed partitions into smaller partitions to avoid differences in partition processing time.

Reroute a query in case of an executor failure.

Collect runtime statistics during query execution.

新問題 44
In which order should the code blocks shown below be run in order to assign articlesDf a DataFrame that lists all items in column attributes ordered by the number of times these items occur, from most to least often?
Sample of DataFrame articlesDf:
1.+——+—————————–+——————-+
2.|itemId|attributes |supplier |
3.+——+—————————–+——————-+
4.|1 |[blue, winter, cozy] |Sports Company Inc.|
5.|2 |[red, summer, fresh, cooling]|YetiX |
6.|3 |[green, summer, travel] |Sports Company Inc.|
7.+——+—————————–+——————-+

1. articlesDf = articlesDf.groupby(“col”)
2. articlesDf = articlesDf.select(explode(col(“attributes”)))
3. articlesDf = articlesDf.orderBy(“count”).select(“col”)
4. articlesDf = articlesDf.sort(“count”,ascending=False).select(“col”)
5. articlesDf = articlesDf.groupby(“col”).count()

4, 5

2, 5, 3

5, 2

2, 3, 4

2, 5, 4

新問題 45
The code block displayed below contains an error. The code block is intended to perform an outer join of DataFrames transactionsDf and itemsDf on columns productId and itemId, respectively.
Find the error.
Code block:
transactionsDf.join(itemsDf, [itemsDf.itemId, transactionsDf.productId], “outer”)

The “outer” argument should be eliminated, since “outer” is the default join type.

The join type needs to be appended to the join() operator, like join().outer() instead of listing it as the last argument inside the join() call.

The term [itemsDf.itemId, transactionsDf.productId] should be replaced by itemsDf.itemId == transactionsDf.productId.

The term [itemsDf.itemId, transactionsDf.productId] should be replaced by itemsDf.col(“itemId”) == transactionsDf.col(“productId”).

The “outer” argument should be eliminated from the call and join should be replaced by joinOuter.

新問題 46
Which of the following statements about storage levels is incorrect?

The cache operator on DataFrames is evaluated like a transformation.

In client mode, DataFrames cached with the MEMORY_ONLY_2 level will not be stored in the edge node’s memory.

Caching can be undone using the DataFrame.unpersist() operator.

MEMORY_AND_DISK replicates cached DataFrames both on memory and disk.

DISK_ONLY will not use the worker node’s memory.

新問題 47
The code block shown below should return a DataFrame with columns transactionsId, predError, value, and f from DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__)

1. filter
2. “transactionId”, “predError”, “value”, “f”

1. select
2. “transactionId, predError, value, f”

1. select
2. [“transactionId”, “predError”, “value”, “f”]

1. where
2. col(“transactionId”), col(“predError”), col(“value”), col(“f”)

1. select
2. col([“transactionId”, “predError”, “value”, “f”])

新問題 48
The code block shown below should return a DataFrame with two columns, itemId and col. In this DataFrame, for each element in column attributes of DataFrame itemDf there should be a separate row in which the column itemId contains the associated itemId from DataFrame itemsDf. The new DataFrame should only contain rows for rows in DataFrame itemsDf in which the column attributes contains the element cozy.
A sample of DataFrame itemsDf is below.
Code block:
itemsDf.__1__(__2__).__3__(__4__, __5__(__6__))

1. filter
2. array_contains(“cozy”)
3. select
4. “itemId”
5. explode
6. “attributes”

1. where
2. “array_contains(attributes, ‘cozy’)”
3. select
4. itemId
5. explode
6. attributes

1. filter
2. “array_contains(attributes, ‘cozy’)”
3. select
4. “itemId”
5. map
6. “attributes”

1. filter
2. “array_contains(attributes, cozy)”
3. select
4. “itemId”
5. explode
6. “attributes”

1. filter
2. “array_contains(attributes, ‘cozy’)”
3. select
4. “itemId”
5. explode
6. “attributes”

新問題 49
Which of the following code blocks creates a new DataFrame with two columns season and wind_speed_ms where column season is of data type string and column wind_speed_ms is of data type double?

spark.DataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]})

spark.createDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”])

1. from pyspark.sql import types as T
2. spark.createDataFrame(((“summer”, 4.5), (“winter”, 7.5)), T.StructType([T.StructField(“season”,

CharType()), T.StructField(“season”, T.DoubleType())]))

spark.newDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”])

spark.createDataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]})

Explanation
spark.createDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”]) Correct. This command uses the Spark Session’s createDataFrame method to create a new DataFrame. Notice how rows, columns, and column names are passed in here: The rows are specified as a Python list. Every entry in the list is a new row. Columns are specified as Python tuples (for example (“summer”, 4.5)). Every column is one entry in the tuple.
The column names are specified as the second argument to createDataFrame(). The documentation (link below) shows that “when schema is a list of column names, the type of each column will be inferred from data” (the first argument). Since values 4.5 and 7.5 are both float variables, Spark will correctly infer the double type for column wind_speed_ms. Given that all values in column
“season” contain only strings, Spark will cast the column appropriately as string.
Find out more about SparkSession.createDataFrame() via the link below.
spark.newDataFrame([(“summer”, 4.5), (“winter”, 7.5)], [“season”, “wind_speed_ms”]) No, the SparkSession does not have a newDataFrame method.
from pyspark.sql import types as T
spark.createDataFrame(((“summer”, 4.5), (“winter”, 7.5)), T.StructType([T.StructField(“season”,
T.CharType()), T.StructField(“season”, T.DoubleType())]))
No. pyspark.sql.types does not have a CharType type. See link below for available data types in Spark.
spark.createDataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]}) No, this is not correct Spark syntax. If you have considered this option to be correct, you may have some experience with Python’s pandas package, in which this would be correct syntax. To create a Spark DataFrame from a Pandas DataFrame, you can simply use spark.createDataFrame(pandasDf) where pandasDf is the Pandas DataFrame.
Find out more about Spark syntax options using the examples in the documentation for SparkSession.createDataFrame linked below.
spark.DataFrame({“season”: [“winter”,”summer”], “wind_speed_ms”: [4.5, 7.5]}) No, the Spark Session (indicated by spark in the code above) does not have a DataFrame method.
More info: pyspark.sql.SparkSession.createDataFrame – PySpark 3.1.1 documentation and Data Types – Spark 3.1.2 Documentation Static notebook | Dynamic notebook: See test 1

新問題 50
The code block displayed below contains one or more errors. The code block should load parquet files at location filePath into a DataFrame, only loading those files that have been modified before
2029-03-20 05:44:46. Spark should enforce a schema according to the schema shown below. Find the error.
Schema:
1.root
2. |– itemId: integer (nullable = true)
3. |– attributes: array (nullable = true)
4. | |– element: string (containsNull = true)
5. |– supplier: string (nullable = true)
Code block:
1.schema = StructType([
2. StructType(“itemId”, IntegerType(), True),
3. StructType(“attributes”, ArrayType(StringType(), True), True),
4. StructType(“supplier”, StringType(), True)
5.])
6.
7.spark.read.options(“modifiedBefore”, “2029-03-20T05:44:46”).schema(schema).load(filePath)

The attributes array is specified incorrectly, Spark cannot identify the file format, and the syntax of the call to Spark’s DataFrameReader is incorrect.

Columns in the schema definition use the wrong object type and the syntax of the call to Spark’s DataFrameReader is incorrect.

The data type of the schema is incompatible with the schema() operator and the modification date threshold is specified incorrectly.

Columns in the schema definition use the wrong object type, the modification date threshold is specified incorrectly, and Spark cannot identify the file format.

Columns in the schema are unable to handle empty values and the modification date threshold is specified incorrectly.

Explanation
Correct code block:
schema = StructType([
StructField(“itemId”, IntegerType(), True),
StructField(“attributes”, ArrayType(StringType(), True), True),
StructField(“supplier”, StringType(), True)
])
spark.read.options(modifiedBefore=”2029-03-20T05:44:46″).schema(schema).parquet(filePath) This question is more difficult than what you would encounter in the exam. In the exam, for this question type, only one error needs to be identified and not “one or multiple” as in the question.
Columns in the schema definition use the wrong object type, the modification date threshold is specified incorrectly, and Spark cannot identify the file format.
Correct! Columns in the schema definition should use the StructField type. Building a schema from pyspark.sql.types, as here using classes like StructType and StructField, is one of multiple ways of expressing a schema in Spark. A StructType always contains a list of StructFields (see documentation linked below). So, nesting StructType and StructType as shown in the question is wrong.
The modification date threshold should be specified by a keyword argument like options(modifiedBefore=”2029-03-20T05:44:46″) and not two consecutive non-keyword arguments as in the original code block (see documentation linked below).
Spark cannot identify the file format correctly, because either it has to be specified by using the DataFrameReader.format(), as an argument to DataFrameReader.load(), or directly by calling, for example, DataFrameReader.parquet().
Columns in the schema are unable to handle empty values and the modification date threshold is specified incorrectly.
No. If StructField would be used for the columns instead of StructType (see above), the third argument specified whether the column is nullable. The original schema shows that columns should be nullable and this is specified correctly by the third argument being True in the schema in the code block.
It is correct, however, that the modification date threshold is specified incorrectly (see above).
The attributes array is specified incorrectly, Spark cannot identify the file format, and the syntax of the call to Spark’s DataFrameReader is incorrect.
Wrong. The attributes array is specified correctly, following the syntax for ArrayType (see linked documentation below). That Spark cannot identify the file format is correct, see correct answer above. In addition, the DataFrameReader is called correctly through the SparkSession spark.
Columns in the schema definition use the wrong object type and the syntax of the call to Spark’s DataFrameReader is incorrect.
Incorrect, the object types in the schema definition are correct and syntax of the call to Spark’s DataFrameReader is correct.
The data type of the schema is incompatible with the schema() operator and the modification date threshold is specified incorrectly.
False. The data type of the schema is StructType and an accepted data type for the DataFrameReader.schema() method. It is correct however that the modification date threshold is specified incorrectly (see correct answer above).

itemsDf.select(explode(“attributes”).alias(“attributes_exploded”)).filter(attributes_exploded.contains(“i”))

itemsDf.explode(attributes).alias(“attributes_exploded”).filter(col(“attributes_exploded”).contains(“i”))

itemsDf.select(explode(“attributes”)).filter(“attributes_exploded”.contains(“i”))

itemsDf.select(explode(“attributes”).alias(“attributes_exploded”)).filter(col(“attributes_exploded”).contain

itemsDf.select(col(“attributes”).explode().alias(“attributes_exploded”)).filter(col(“attributes_exploded”).co

新問題 52
In which order should the code blocks shown below be run in order to return the number of records that are not empty in column value in the DataFrame resulting from an inner join of DataFrame transactionsDf and itemsDf on columns productId and itemId, respectively?
1. .filter(~isnull(col(‘value’)))
2. .count()
3. transactionsDf.join(itemsDf, col(“transactionsDf.productId”)==col(“itemsDf.itemId”))
4. transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId, how=’inner’)
5. .filter(col(‘value’).isnotnull())
6. .sum(col(‘value’))

4, 1, 2

3, 1, 6

3, 1, 2

3, 5, 2

4, 6

新問題 53
Which of the following code blocks performs an inner join of DataFrames transactionsDf and itemsDf on columns productId and itemId, respectively, excluding columns value and storeId from DataFrame transactionsDf and column attributes from DataFrame itemsDf?

transactionsDf.drop(‘value’, ‘storeId’).join(itemsDf.select(‘attributes’), transactionsDf.productId==itemsDf.itemId)

1.transactionsDf.createOrReplaceTempView(‘transactionsDf’)
2.itemsDf.createOrReplaceTempView(‘itemsDf’)
3.
4.spark.sql(“SELECT -value, -storeId FROM transactionsDf INNER JOIN itemsDf ON productId==itemId”).drop(“attributes”)

transactionsDf.drop(“value”, “storeId”).join(itemsDf.drop(“attributes”),
“transactionsDf.productId==itemsDf.itemId”)

1.transactionsDf
2. .drop(col(‘value’), col(‘storeId’))
3. .join(itemsDf.drop(col(‘attributes’)), col(‘productId’)==col(‘itemId’))

1.transactionsDf.createOrReplaceTempView(‘transactionsDf’)
2.itemsDf.createOrReplaceTempView(‘itemsDf’)
3.
4.statement = “””
5.SELECT * FROM transactionsDf
6.INNER JOIN itemsDf
7.ON transactionsDf.productId==itemsDf.itemId
8.”””
9.spark.sql(statement).drop(“value”, “storeId”, “attributes”)

Explanation
This question offers you a wide variety of answers for a seemingly simple question. However, this variety reflects the variety of ways that one can express a join in PySpark. You need to understand some SQL syntax to get to the correct answer here.
transactionsDf.createOrReplaceTempView(‘transactionsDf’)
itemsDf.createOrReplaceTempView(‘itemsDf’)
statement = “””
SELECT * FROM transactionsDf
INNER JOIN itemsDf
ON transactionsDf.productId==itemsDf.itemId
“””
spark.sql(statement).drop(“value”, “storeId”, “attributes”)
Correct – this answer uses SQL correctly to perform the inner join and afterwards drops the unwanted columns. This is totally fine. If you are unfamiliar with the triple-quote “”” in Python: This allows you to express strings as multiple lines.
transactionsDf \
drop(col(‘value’), col(‘storeId’)) \
join(itemsDf.drop(col(‘attributes’)), col(‘productId’)==col(‘itemId’))
No, this answer option is a trap, since DataFrame.drop() does not accept a list of Column objects. You could use transactionsDf.drop(‘value’, ‘storeId’) instead.
transactionsDf.drop(“value”, “storeId”).join(itemsDf.drop(“attributes”),
“transactionsDf.productId==itemsDf.itemId”)
Incorrect – Spark does not evaluate “transactionsDf.productId==itemsDf.itemId” as a valid join expression.
This would work if it would not be a string.
transactionsDf.drop(‘value’, ‘storeId’).join(itemsDf.select(‘attributes’), transactionsDf.productId==itemsDf.itemId) Wrong, this statement incorrectly uses itemsDf.select instead of itemsDf.drop.
transactionsDf.createOrReplaceTempView(‘transactionsDf’)
itemsDf.createOrReplaceTempView(‘itemsDf’)
spark.sql(“SELECT -value, -storeId FROM transactionsDf INNER JOIN itemsDf ON productId==itemId”).drop(“attributes”) No, here the SQL expression syntax is incorrect. Simply specifying -columnName does not drop a column.
More info: pyspark.sql.DataFrame.join – PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

新問題 54
The code block displayed below contains an error. The code block should produce a DataFrame with color as the only column and three rows with color values of red, blue, and green, respectively.
Find the error.
Code block:
1.spark.createDataFrame([(“red”,), (“blue”,), (“green”,)], “color”)
Instead of calling spark.createDataFrame, just DataFrame should be called.

The commas in the tuples with the colors should be eliminated.

The colors red, blue, and green should be expressed as a simple Python list, and not a list of tuples.

Instead of color, a data type should be specified.

The “color” expression needs to be wrapped in brackets, so it reads [“color”].

新問題 55
The code block displayed below contains an error. When the code block below has executed, it should have divided DataFrame transactionsDf into 14 parts, based on columns storeId and transactionDate (in this order). Find the error.
Code block:
transactionsDf.coalesce(14, (“storeId”, “transactionDate”))

The parentheses around the column names need to be removed and .select() needs to be appended to the code block.

Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block.
(Correct)

Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .select() needs to be appended to the code block.

Operator coalesce needs to be replaced by repartition and the parentheses around the column names need to be replaced by square brackets.

Operator coalesce needs to be replaced by repartition.

新問題 56
Which of the following describes a narrow transformation?

narrow transformation is an operation in which data is exchanged across partitions.

A narrow transformation is a process in which data from multiple RDDs is used.

A narrow transformation is a process in which 32-bit float variables are cast to smaller float variables, like 16-bit or 8-bit float variables.

A narrow transformation is an operation in which data is exchanged across the cluster.

A narrow transformation is an operation in which no data is exchanged across the cluster.

新問題 57
Which of the following describes Spark’s Adaptive Query Execution?

Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.

Adaptive Query Execution is enabled in Spark by default.

Adaptive Query Execution reoptimizes queries at execution points.

Adaptive Query Execution features are dynamically switching join strategies and dynamically optimizing skew joins.

Adaptive Query Execution applies to all kinds of queries.

新問題 58
The code block displayed below contains an error. The code block should display the schema of DataFrame transactionsDf. Find the error.
Code block:
transactionsDf.rdd.printSchema

There is no way to print a schema directly in Spark, since the schema can be printed easily through using print(transactionsDf.columns), so that should be used instead.

The code block should be wrapped into a print() operation.

printSchema is only accessible through the spark session, so the code block should be rewritten as spark.printSchema(transactionsDf).

printSchema is a method and should be written as printSchema(). It is also not callable through transactionsDf.rdd, but should be called directly from transactionsDf.
(Correct)

printSchema is a not a method of transactionsDf.rdd. Instead, the schema should be printed via transactionsDf.print_schema().

新問題 59
The code block shown below should return an exact copy of DataFrame transactionsDf that does not include rows in which values in column storeId have the value 25. Choose the answer that correctly fills the blanks in the code block to accomplish this.

transactionsDf.remove(transactionsDf.storeId==25)

transactionsDf.where(transactionsDf.storeId!=25)

transactionsDf.filter(transactionsDf.storeId==25)

transactionsDf.drop(transactionsDf.storeId==25)

transactionsDf.select(transactionsDf.storeId!=25)

新問題 60
Which of the following code blocks removes all rows in the 6-column DataFrame transactionsDf that have missing data in at least 3 columns?

transactionsDf.dropna(“any”)

transactionsDf.dropna(thresh=4)

transactionsDf.drop.na(“”,2)

transactionsDf.dropna(thresh=2)

transactionsDf.dropna(“”,4)

新問題 61
Which of the following is one of the big performance advantages that Spark has over Hadoop?

Spark achieves great performance by storing data in the DAG format, whereas Hadoop can only use parquet files.

Spark achieves higher resiliency for queries since, different from Hadoop, it can be deployed on Kubernetes.

Spark achieves great performance by storing data and performing computation in memory, whereas large jobs in Hadoop require a large amount of relatively slow disk I/O operations.

Spark achieves great performance by storing data in the HDFS format, whereas Hadoop can only use parquet files.

Spark achieves performance gains for developers by extending Hadoop’s DataFrames with a user-friendly API.

新問題 62
Which of the following code blocks returns a new DataFrame with only columns predError and values of every second row of DataFrame transactionsDf?
Entire DataFrame transactionsDf:
1.+————-+———+—–+——-+———+—-+
2.|transactionId|predError|value|storeId|productId| f|
3.+————-+———+—–+——-+———+—-+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.| 4| null| null| 3| 2|null|
8.| 5| null| null| null| 2|null|
9.| 6| 3| 2| 25| 2|null|
10.+————-+———+—–+——-+———+—-+

transactionsDf.filter(col(“transactionId”).isin([3,4,6])).select([predError, value])

transactionsDf.select(col(“transactionId”).isin([3,4,6]), “predError”, “value”)

transactionsDf.filter(“transactionId” % 2 == 0).select(“predError”, “value”)

transactionsDf.filter(col(“transactionId”) % 2 == 0).select(“predError”, “value”) (Correct)

1.transactionsDf.createOrReplaceTempView(“transactionsDf”)
2.spark.sql(“FROM transactionsDf SELECT predError, value WHERE transactionId % 2 = 2”)

transactionsDf.filter(col(transactionId).isin([3,4,6]))

新問題 63
Which of the following statements about garbage collection in Spark is incorrect?

Garbage collection information can be accessed in the Spark UI’s stage detail view.

Optimizing garbage collection performance in Spark may limit caching ability.

Manually persisting RDDs in Spark prevents them from being garbage collected.

In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector.

Serialized caching is a strategy to increase the performance of garbage collection.

Explanation
Manually persisting RDDs in Spark prevents them from being garbage collected.
This statement is incorrect, and thus the correct answer to the question. Spark’s garbage collector will remove even persisted objects, albeit in an “LRU” fashion. LRU stands for least recently used.
So, during a garbage collection run, the objects that were used the longest time ago will be garbage collected first.
See the linked StackOverflow post below for more information.
Serialized caching is a strategy to increase the performance of garbage collection.
This statement is correct. The more Java objects Spark needs to collect during garbage collection, the longer it takes. Storing a collection of many Java objects, such as a DataFrame with a complex schema, through serialization as a single byte array thus increases performance. This means that garbage collection takes less time on a serialized DataFrame than an unserialized DataFrame.
Optimizing garbage collection performance in Spark may limit caching ability.
This statement is correct. A full garbage collection run slows down a Spark application. When taking about
“tuning” garbage collection, we mean reducing the amount or duration of these slowdowns.
A full garbage collection run is triggered when the Old generation of the Java heap space is almost full. (If you are unfamiliar with this concept, check out the link to the Garbage Collection Tuning docs below.) Thus, one measure to avoid triggering a garbage collection run is to prevent the Old generation share of the heap space to be almost full.
To achieve this, one may decrease its size. Objects with sizes greater than the Old generation space will then be discarded instead of cached (stored) in the space and helping it to be “almost full”.
This will decrease the number of full garbage collection runs, increasing overall performance.
Inevitably, however, objects will need to be recomputed when they are needed. So, this mechanism only works when a Spark application needs to reuse cached data as little as possible.
Garbage collection information can be accessed in the Spark UI’s stage detail view.
This statement is correct. The task table in the Spark UI’s stage detail view has a “GC Time” column, indicating the garbage collection time needed per task.
In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector.
This statement is correct. The G1 garbage collector, also known as garbage first garbage collector, is an alternative to the default Parallel garbage collector.
While the default Parallel garbage collector divides the heap into a few static regions, the G1 garbage collector divides the heap into many small regions that are created dynamically. The G1 garbage collector has certain advantages over the Parallel garbage collector which improve performance particularly for Spark workloads that require high throughput and low latency.
The G1 garbage collector is not enabled by default, and you need to explicitly pass an argument to Spark to enable it. For more information about the two garbage collectors, check out the Databricks article linked below.

Associate-Developer-Apache-Spark問題集で2022年最新のDatabricks Associate-Developer-Apache-Spark試験問題 [Q40-Q63]

コメントを残すコメントをキャンセル

関連認証

Associate-Developer-Apache-Spark 練習テスト

最近の投稿

アーカイブ

カテゴリー

Associate-Developer-Apache-Spark問題集で2022年最新のDatabricks Associate-Developer-Apache-Spark試験問題 [Q40-Q63]

コメントを残す コメントをキャンセル

関連認証

Associate-Developer-Apache-Spark 練習テスト

最近の投稿

アーカイブ

カテゴリー

コメントを残すコメントをキャンセル